Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X......X.....] duration: 41.820534436s trace: https://jaeger.dev.storj.io/trace/2340efa351101f24 FAIL (2025-12-26 09:49:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 26 09:44:00.880088265 (498h15m0.751171301s) [noise=true , address=45.91.116.7:4016 , piece_id=JAZZQAOSDSXEAJS57ZNGRUE4RLXQQ6MPBDDJGPD4QNSLU56TZYSQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.880229845 (498h15m0.75131288s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.880757570 (498h15m0.751840615s) [noise=true , address=145.79.227.30:4765 , piece_id=IJRWZ3YBUSGZJ3WJKXEEOWWIFMO25CMBYR2PO537EKHTQHB3XP2Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.881224599 (498h15m0.752307644s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.882867498 (498h15m0.753950533s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.883629057 (498h15m0.754712102s) [noise=true , address=45.91.116.7:4042 , piece_id=QOYKGPE2E7SSX7YCPVHMZYVXOH43MW5ODEJMYEACSUDLPJLP6QHA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.884590658 (498h15m0.755673734s) [noise=true , address=45.91.116.9:4272 , piece_id=4LG7DWD5GJEXX4KGRS6QROVWBXGCUZ4LWOBMIPI2AEK2Q3NIK7XQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.884836046 (498h15m0.755919081s) [noise=true , address=45.91.116.10:4104 , piece_id=5YHM2XVPLW5BUDOFKSRPAOU6FAQEZUFVJQU2PKAXMLSGFBC3L73Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.886094181 (498h15m0.757177236s) [noise=true , address=145.79.227.7:4322 , piece_id=4IXKAMY6AB6HRGNMZLLMZ2WWUUHXRGZGD35VZMORUVG4UHV2Q4PA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.886114190 (498h15m0.757197235s) [noise=true , address=45.91.116.4:4944 , piece_id=PLIUZEYKODIE3JKUFVI43FZ47DYVAE6CGFXT45N6MBOLYJV626UQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.887478658 (498h15m0.758561693s) [noise=true , address=77.90.31.12:4598 , piece_id=L33QFG7QEALMVTS2IOF7DZOA4LCFMKKRPG4E77EALQK2NG622Q2Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.887528633 (498h15m0.758611668s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.888256428 (498h15m0.759339474s) [noise=true , address=45.91.116.14:4195 , piece_id=QEOVODBM24X6ROM2ME3GCTPVUHFHDE5JBWAQSECCRITOAJIFR4SA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.888915033 (498h15m0.759998078s) [noise=true , address=145.79.227.44:4203 , piece_id=6XBUBLM3UGODNA2INBVZLSKVQSQIVKECAYEFMHULVHI4FN7LHMKQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.890072357 (498h15m0.761155392s) [noise=true , address=45.91.116.11:4129 , piece_id=QCGCBLS7PHXGRAB2ZGPUDKL5ADSDMCZHHSJPQVOSPHXMX7O6UIJA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.890947614 (498h15m0.762030659s) [noise=true , address=77.90.31.14:4675 , piece_id=BHLZL54ETYKAACJYLGD3JGOJEIM7IVOS4BIIKDN3ZHDEDBVQBMQA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:02.349863155 (498h15m2.22094619s) [upload=437299]: Next returned nil segment Dec 26 09:44:02.349900766 (498h15m2.220983811s) [upload=437299]: Waiting for error group managing segments... Dec 26 09:44:02.781836534 (498h15m2.65291958s) [noise=true , address=145.79.227.7:4322 , piece_id=4IXKAMY6AB6HRGNMZLLMZ2WWUUHXRGZGD35VZMORUVG4UHV2Q4PA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:02.928532198 (498h15m2.799615243s) [noise=true , address=145.79.227.34:4937 , piece_id=RRTHSCYGQXVBGWD4IATI7XEBM32RDPVQR7T4J3XJQXUIGWU45GCA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:02.959038798 (498h15m2.830121853s) [noise=true , address=77.90.31.14:4675 , piece_id=BHLZL54ETYKAACJYLGD3JGOJEIM7IVOS4BIIKDN3ZHDEDBVQBMQA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.052394553 (498h15m2.923477598s) [noise=true , address=77.90.31.12:4598 , piece_id=L33QFG7QEALMVTS2IOF7DZOA4LCFMKKRPG4E77EALQK2NG622Q2Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.066899663 (498h15m2.937982708s) [noise=true , address=45.91.116.4:4944 , piece_id=PLIUZEYKODIE3JKUFVI43FZ47DYVAE6CGFXT45N6MBOLYJV626UQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.081280326 (498h15m2.952363381s) [noise=true , address=45.91.116.10:4104 , piece_id=5YHM2XVPLW5BUDOFKSRPAOU6FAQEZUFVJQU2PKAXMLSGFBC3L73Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.096434834 (498h15m2.967517869s) [noise=true , address=45.91.116.7:4016 , piece_id=JAZZQAOSDSXEAJS57ZNGRUE4RLXQQ6MPBDDJGPD4QNSLU56TZYSQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.107137242 (498h15m2.978220287s) [noise=true , address=45.91.116.14:4195 , piece_id=QEOVODBM24X6ROM2ME3GCTPVUHFHDE5JBWAQSECCRITOAJIFR4SA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.176790474 (498h15m3.047873519s) [noise=true , address=145.79.227.40:4074 , piece_id=SPEIB4FDHK7I45NIXB3CLX5HNVNKSQZ5APXE4ULGW3MVPEEEKCOA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.431315263 (498h15m3.302398308s) [noise=true , address=45.91.116.9:4272 , piece_id=4LG7DWD5GJEXX4KGRS6QROVWBXGCUZ4LWOBMIPI2AEK2Q3NIK7XQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.710113361 (498h15m3.581196406s) [noise=true , address=145.79.227.44:4203 , piece_id=6XBUBLM3UGODNA2INBVZLSKVQSQIVKECAYEFMHULVHI4FN7LHMKQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.716995455 (498h15m3.5880785s) [noise=true , address=145.79.227.30:4765 , piece_id=IJRWZ3YBUSGZJ3WJKXEEOWWIFMO25CMBYR2PO537EKHTQHB3XP2Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.183858151 (498h15m4.054941206s) [noise=true , address=45.91.116.11:4129 , piece_id=QCGCBLS7PHXGRAB2ZGPUDKL5ADSDMCZHHSJPQVOSPHXMX7O6UIJA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.462428025 (498h15m4.33351106s) [noise=true , address=45.91.116.7:4042 , piece_id=QOYKGPE2E7SSX7YCPVHMZYVXOH43MW5ODEJMYEACSUDLPJLP6QHA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.462455627 (498h15m4.333538662s) [seg_pos={0 0}, upload=437299]: Segment reached optimal threshold of 14 pieces. Dec 26 09:44:04.462751150 (498h15m4.333834195s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection Dec 26 09:44:04.462761229 (498h15m4.333844284s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463065669 (498h15m4.334148724s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463124842 (498h15m4.334207887s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463174215 (498h15m4.334257361s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FNXVNAv3KeiRqTnsvQkMobyRcNFmPAAjVqSNvgs6MPJH4smVU): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463187411 (498h15m4.334270456s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463448007 (498h15m4.334531052s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463458536 (498h15m4.334541582s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463503002 (498h15m4.334586047s) [seg_pos={0 0}, upload=437299]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 09:44:04.463624553 (498h15m4.334707598s) [upload=437299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:44:04.463631466 (498h15m4.334714521s) [upload=437299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:44:04.753033747 (498h15m4.624116782s) [upload=437299]: Done uploading segments... Dec 26 09:44:04.753119550 (498h15m4.624202595s) [upload=437299]: Upload finished. err: <nil> Duration: 41.820562129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX..X.......X.....XX.XX.XXX.....X..X.....X.X...X..XXXXXXX...] duration: 7.225824829s trace: https://jaeger.dev.storj.io/trace/5961db0f7939a572 FAIL (2025-12-26 09:48:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 6R5AHGWPFRAZW3442VDKUX5ILHSJF7GMT6OTOU75O5E7PFV44K7Q) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: T2E2THTWPZZ2KNYZBRL2RAINUDJK3MOCZHMVWVG7EBPIJCOA5SRQ) context deadline exceeded ======LOGS====== Dec 26 09:46:08.165062995 (498h17m8.03614605s) [upload=437335]: Starting upload Dec 26 09:46:08.165149940 (498h17m8.036232985s) [upload=437335]: Done starting upload Dec 26 09:46:08.165273245 (498h17m8.03635629s) [upload=437335]: Uploading segments... Dec 26 09:46:08.165437147 (498h17m8.036520192s) [upload=437335]: Got next segment. Inline: false Dec 26 09:46:08.165478266 (498h17m8.036561311s) [upload=437335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:46:08.165483325 (498h17m8.03656637s) [upload=437335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:46:08.414578364 (498h17m8.285661399s) [seg_pos={0 0}, upload=437335]: Begin upload segment... Dec 26 09:46:08.414622609 (498h17m8.285705644s) [seg_pos={0 0}, upload=437335]: Done begin upload segment. Dec 26 09:46:08.414648839 (498h17m8.285731884s) [noise=true , address=145.79.227.17:4970 , piece_id=Q6QVMEARDXMYYLHXZVSNZYMPD2J4GQUJT7FZH6X5CZPGRJ2CR2HA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.414740844 (498h17m8.285823879s) [noise=true , address=45.91.116.6:4973 , piece_id=ZGLKRRFLHU6YYEJ2ML23D6LWU5NPQPABC34L5CEHDA66JTASJ5GA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.415427883 (498h17m8.286510918s) [noise=true , address=145.79.227.36:4517 , piece_id=BD3YHDVEDPBNALVN32SLT6DZJPDBOS2TSM5TW7YT72WJUY7ZR7JA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.415816282 (498h17m8.286899317s) [noise=true , address=145.79.227.41:4086 , piece_id=22EPMHWC2W3R5ZUJODJXBEPZVN35NTPCA7N2ZTAMF5OPZXETQP5Q, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.416313959 (498h17m8.287396995s) [noise=true , address=145.79.227.45:4230 , piece_id=67GCDFMYPGATGWD7SEKQVMEB65XARZ3EF7O62SHFVGZGZMAE42JQ, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.416644099 (498h17m8.287727134s) [noise=true , address=145.79.227.29:4941 , piece_id=FHAHQWT3ANQOCQYMQHW7X6QLHGFJLZYACNYGS2OW4NNGJXNF3QFQ, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.416941124 (498h17m8.288024159s) [noise=true , address=145.79.227.2:4115 , piece_id=HTRB5757CCREXIYVDK7MS2T6VIYBATJ6VMMJILH353VN4KELEMTA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.417223171 (498h17m8.288306206s) [noise=true , address=77.90.31.13:4629 , piece_id=6R5AHGWPFRAZW3442VDKUX5ILHSJF7GMT6OTOU75O5E7PFV44K7Q, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.417555835 (498h17m8.28863887s) [noise=true , address=145.79.227.42:4123 , piece_id=JAOPHXKNAJOBIEK4RYA2OWEUOAMP462WJQJJFEWSYCRK62R5CJRA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.418732335 (498h17m8.28981537s) [noise=true , address=45.91.116.5:4300 , piece_id=SMKSHPROTHQ26ZNVUAVIMSX5DU5ZAD5V6TRPARSWZZBCOORTT5XQ, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.418885456 (498h17m8.289968491s) [noise=true , address=45.91.116.11:4120 , piece_id=T2E2THTWPZZ2KNYZBRL2RAINUDJK3MOCZHMVWVG7EBPIJCOA5SRQ, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.419814956 (498h17m8.290898001s) [noise=true , address=45.91.116.14:4190 , piece_id=D7TATQKJASAAGGVPYRW3HH26R4DNQO3H7XNSNO6HGCVVBJTS554Q, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.420438995 (498h17m8.29152204s) [noise=true , address=45.91.116.4:4940 , piece_id=MEW6HWSNGGOPG7ELLDJLOPU7AJJ5Z7ERMNPRA3GCZKXR5RVYU6UA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.421883214 (498h17m8.292966259s) [noise=true , address=145.79.227.46:4603 , piece_id=XXEBZ4TQ2MGMDL3F5KYTZIZVA6HN376MKW5IKSITC2ED5YIMA5EA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.423567993 (498h17m8.294651028s) [noise=true , address=145.79.227.39:4015 , piece_id=YKJBRUPAJLX64MV4YS4WUG2LWOZA42WEF4BOGQYMGMRXRHSZUIKA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.425066296 (498h17m8.296149331s) [noise=true , address=145.79.227.37:4566 , piece_id=YNVC2E7IYVDAKT6QTBLTXV35Q2P73DFDWXCZ4LULV4PJ4XUSKE5Q, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:10.138884602 (498h17m10.009967658s) [upload=437335]: Next returned nil segment Dec 26 09:46:10.138909960 (498h17m10.009992995s) [upload=437335]: Waiting for error group managing segments... Dec 26 09:46:10.413906521 (498h17m10.284989576s) [noise=true , address=145.79.227.37:4566 , piece_id=YNVC2E7IYVDAKT6QTBLTXV35Q2P73DFDWXCZ4LULV4PJ4XUSKE5Q, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.427869810 (498h17m10.298952865s) [noise=true , address=145.79.227.45:4230 , piece_id=67GCDFMYPGATGWD7SEKQVMEB65XARZ3EF7O62SHFVGZGZMAE42JQ, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.430136766 (498h17m10.301219801s) [noise=true , address=145.79.227.29:4941 , piece_id=FHAHQWT3ANQOCQYMQHW7X6QLHGFJLZYACNYGS2OW4NNGJXNF3QFQ, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.430493807 (498h17m10.301576842s) [noise=true , address=145.79.227.41:4086 , piece_id=22EPMHWC2W3R5ZUJODJXBEPZVN35NTPCA7N2ZTAMF5OPZXETQP5Q, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.434724542 (498h17m10.305807587s) [noise=true , address=145.79.227.39:4015 , piece_id=YKJBRUPAJLX64MV4YS4WUG2LWOZA42WEF4BOGQYMGMRXRHSZUIKA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.435101160 (498h17m10.306184195s) [noise=true , address=145.79.227.46:4603 , piece_id=XXEBZ4TQ2MGMDL3F5KYTZIZVA6HN376MKW5IKSITC2ED5YIMA5EA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.443912207 (498h17m10.314995252s) [noise=true , address=145.79.227.17:4970 , piece_id=Q6QVMEARDXMYYLHXZVSNZYMPD2J4GQUJT7FZH6X5CZPGRJ2CR2HA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.444769119 (498h17m10.315852164s) [noise=true , address=145.79.227.42:4123 , piece_id=JAOPHXKNAJOBIEK4RYA2OWEUOAMP462WJQJJFEWSYCRK62R5CJRA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.446579456 (498h17m10.317662502s) [noise=true , address=145.79.227.36:4517 , piece_id=BD3YHDVEDPBNALVN32SLT6DZJPDBOS2TSM5TW7YT72WJUY7ZR7JA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.459713005 (498h17m10.33079605s) [noise=true , address=145.79.227.2:4115 , piece_id=HTRB5757CCREXIYVDK7MS2T6VIYBATJ6VMMJILH353VN4KELEMTA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.668029625 (498h17m10.53911266s) [noise=true , address=45.91.116.11:4120 , piece_id=T2E2THTWPZZ2KNYZBRL2RAINUDJK3MOCZHMVWVG7EBPIJCOA5SRQ, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.732776044 (498h17m10.603859079s) [noise=true , address=77.90.31.13:4629 , piece_id=6R5AHGWPFRAZW3442VDKUX5ILHSJF7GMT6OTOU75O5E7PFV44K7Q, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.765684757 (498h17m10.636767802s) [noise=true , address=45.91.116.6:4973 , piece_id=ZGLKRRFLHU6YYEJ2ML23D6LWU5NPQPABC34L5CEHDA66JTASJ5GA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.765707080 (498h17m10.636790115s) [seg_pos={0 0}, upload=437335]: Segment reached optimal threshold of 13 pieces. Dec 26 09:46:10.766046667 (498h17m10.637129702s) [noise=true , address=45.91.116.14:4190 , piece_id=D7TATQKJASAAGGVPYRW3HH26R4DNQO3H7XNSNO6HGCVVBJTS554Q, seg_pos={0 0}, upload=437335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 26 09:46:10.766054642 (498h17m10.637137677s) [noise=true , address=45.91.116.14:4190 , piece_id=D7TATQKJASAAGGVPYRW3HH26R4DNQO3H7XNSNO6HGCVVBJTS554Q, seg_pos={0 0}, upload=437335]: Long tail cancellation due to optimal threshold Dec 26 09:46:10.766348301 (498h17m10.637431336s) [noise=true , address=45.91.116.5:4300 , piece_id=SMKSHPROTHQ26ZNVUAVIMSX5DU5ZAD5V6TRPARSWZZBCOORTT5XQ, seg_pos={0 0}, upload=437335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:51552->45.91.116.5:4300: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 09:46:10.766354824 (498h17m10.637437869s) [noise=true , address=45.91.116.5:4300 , piece_id=SMKSHPROTHQ26ZNVUAVIMSX5DU5ZAD5V6TRPARSWZZBCOORTT5XQ, seg_pos={0 0}, upload=437335]: Long tail cancellation due to optimal threshold Dec 26 09:46:10.766548141 (498h17m10.637631176s) [noise=true , address=45.91.116.4:4940 , piece_id=MEW6HWSNGGOPG7ELLDJLOPU7AJJ5Z7ERMNPRA3GCZKXR5RVYU6UA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: read tcp 188.166.225.170:53780->45.91.116.4:4940: use of closed network connection; context canceled Dec 26 09:46:10.766557589 (498h17m10.637640634s) [noise=true , address=45.91.116.4:4940 , piece_id=MEW6HWSNGGOPG7ELLDJLOPU7AJJ5Z7ERMNPRA3GCZKXR5RVYU6UA, seg_pos={0 0}, upload=437335]: Long tail cancellation due to optimal threshold Dec 26 09:46:10.766572508 (498h17m10.637655543s) [seg_pos={0 0}, upload=437335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:46:10.766639335 (498h17m10.637722381s) [upload=437335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:46:10.766651428 (498h17m10.637734463s) [upload=437335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:46:11.066893972 (498h17m10.937977057s) [upload=437335]: Done uploading segments... Dec 26 09:46:11.066930682 (498h17m10.938013767s) [upload=437335]: Upload finished. err: <nil> Duration: 7.225852631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X......X....] duration: 53.913772685s trace: https://jaeger.dev.storj.io/trace/2340efa351101f24 FAIL (2025-12-26 09:48:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 26 09:44:00.880088265 (498h15m0.751171301s) [noise=true , address=45.91.116.7:4016 , piece_id=JAZZQAOSDSXEAJS57ZNGRUE4RLXQQ6MPBDDJGPD4QNSLU56TZYSQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.880229845 (498h15m0.75131288s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.880757570 (498h15m0.751840615s) [noise=true , address=145.79.227.30:4765 , piece_id=IJRWZ3YBUSGZJ3WJKXEEOWWIFMO25CMBYR2PO537EKHTQHB3XP2Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.881224599 (498h15m0.752307644s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.882867498 (498h15m0.753950533s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.883629057 (498h15m0.754712102s) [noise=true , address=45.91.116.7:4042 , piece_id=QOYKGPE2E7SSX7YCPVHMZYVXOH43MW5ODEJMYEACSUDLPJLP6QHA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.884590658 (498h15m0.755673734s) [noise=true , address=45.91.116.9:4272 , piece_id=4LG7DWD5GJEXX4KGRS6QROVWBXGCUZ4LWOBMIPI2AEK2Q3NIK7XQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.884836046 (498h15m0.755919081s) [noise=true , address=45.91.116.10:4104 , piece_id=5YHM2XVPLW5BUDOFKSRPAOU6FAQEZUFVJQU2PKAXMLSGFBC3L73Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.886094181 (498h15m0.757177236s) [noise=true , address=145.79.227.7:4322 , piece_id=4IXKAMY6AB6HRGNMZLLMZ2WWUUHXRGZGD35VZMORUVG4UHV2Q4PA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.886114190 (498h15m0.757197235s) [noise=true , address=45.91.116.4:4944 , piece_id=PLIUZEYKODIE3JKUFVI43FZ47DYVAE6CGFXT45N6MBOLYJV626UQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.887478658 (498h15m0.758561693s) [noise=true , address=77.90.31.12:4598 , piece_id=L33QFG7QEALMVTS2IOF7DZOA4LCFMKKRPG4E77EALQK2NG622Q2Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.887528633 (498h15m0.758611668s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.888256428 (498h15m0.759339474s) [noise=true , address=45.91.116.14:4195 , piece_id=QEOVODBM24X6ROM2ME3GCTPVUHFHDE5JBWAQSECCRITOAJIFR4SA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.888915033 (498h15m0.759998078s) [noise=true , address=145.79.227.44:4203 , piece_id=6XBUBLM3UGODNA2INBVZLSKVQSQIVKECAYEFMHULVHI4FN7LHMKQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.890072357 (498h15m0.761155392s) [noise=true , address=45.91.116.11:4129 , piece_id=QCGCBLS7PHXGRAB2ZGPUDKL5ADSDMCZHHSJPQVOSPHXMX7O6UIJA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.890947614 (498h15m0.762030659s) [noise=true , address=77.90.31.14:4675 , piece_id=BHLZL54ETYKAACJYLGD3JGOJEIM7IVOS4BIIKDN3ZHDEDBVQBMQA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:02.349863155 (498h15m2.22094619s) [upload=437299]: Next returned nil segment Dec 26 09:44:02.349900766 (498h15m2.220983811s) [upload=437299]: Waiting for error group managing segments... Dec 26 09:44:02.781836534 (498h15m2.65291958s) [noise=true , address=145.79.227.7:4322 , piece_id=4IXKAMY6AB6HRGNMZLLMZ2WWUUHXRGZGD35VZMORUVG4UHV2Q4PA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:02.928532198 (498h15m2.799615243s) [noise=true , address=145.79.227.34:4937 , piece_id=RRTHSCYGQXVBGWD4IATI7XEBM32RDPVQR7T4J3XJQXUIGWU45GCA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:02.959038798 (498h15m2.830121853s) [noise=true , address=77.90.31.14:4675 , piece_id=BHLZL54ETYKAACJYLGD3JGOJEIM7IVOS4BIIKDN3ZHDEDBVQBMQA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.052394553 (498h15m2.923477598s) [noise=true , address=77.90.31.12:4598 , piece_id=L33QFG7QEALMVTS2IOF7DZOA4LCFMKKRPG4E77EALQK2NG622Q2Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.066899663 (498h15m2.937982708s) [noise=true , address=45.91.116.4:4944 , piece_id=PLIUZEYKODIE3JKUFVI43FZ47DYVAE6CGFXT45N6MBOLYJV626UQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.081280326 (498h15m2.952363381s) [noise=true , address=45.91.116.10:4104 , piece_id=5YHM2XVPLW5BUDOFKSRPAOU6FAQEZUFVJQU2PKAXMLSGFBC3L73Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.096434834 (498h15m2.967517869s) [noise=true , address=45.91.116.7:4016 , piece_id=JAZZQAOSDSXEAJS57ZNGRUE4RLXQQ6MPBDDJGPD4QNSLU56TZYSQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.107137242 (498h15m2.978220287s) [noise=true , address=45.91.116.14:4195 , piece_id=QEOVODBM24X6ROM2ME3GCTPVUHFHDE5JBWAQSECCRITOAJIFR4SA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.176790474 (498h15m3.047873519s) [noise=true , address=145.79.227.40:4074 , piece_id=SPEIB4FDHK7I45NIXB3CLX5HNVNKSQZ5APXE4ULGW3MVPEEEKCOA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.431315263 (498h15m3.302398308s) [noise=true , address=45.91.116.9:4272 , piece_id=4LG7DWD5GJEXX4KGRS6QROVWBXGCUZ4LWOBMIPI2AEK2Q3NIK7XQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.710113361 (498h15m3.581196406s) [noise=true , address=145.79.227.44:4203 , piece_id=6XBUBLM3UGODNA2INBVZLSKVQSQIVKECAYEFMHULVHI4FN7LHMKQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.716995455 (498h15m3.5880785s) [noise=true , address=145.79.227.30:4765 , piece_id=IJRWZ3YBUSGZJ3WJKXEEOWWIFMO25CMBYR2PO537EKHTQHB3XP2Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.183858151 (498h15m4.054941206s) [noise=true , address=45.91.116.11:4129 , piece_id=QCGCBLS7PHXGRAB2ZGPUDKL5ADSDMCZHHSJPQVOSPHXMX7O6UIJA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.462428025 (498h15m4.33351106s) [noise=true , address=45.91.116.7:4042 , piece_id=QOYKGPE2E7SSX7YCPVHMZYVXOH43MW5ODEJMYEACSUDLPJLP6QHA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.462455627 (498h15m4.333538662s) [seg_pos={0 0}, upload=437299]: Segment reached optimal threshold of 14 pieces. Dec 26 09:44:04.462751150 (498h15m4.333834195s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection Dec 26 09:44:04.462761229 (498h15m4.333844284s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463065669 (498h15m4.334148724s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463124842 (498h15m4.334207887s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463174215 (498h15m4.334257361s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FNXVNAv3KeiRqTnsvQkMobyRcNFmPAAjVqSNvgs6MPJH4smVU): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463187411 (498h15m4.334270456s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463448007 (498h15m4.334531052s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463458536 (498h15m4.334541582s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463503002 (498h15m4.334586047s) [seg_pos={0 0}, upload=437299]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 09:44:04.463624553 (498h15m4.334707598s) [upload=437299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:44:04.463631466 (498h15m4.334714521s) [upload=437299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:44:04.753033747 (498h15m4.624116782s) [upload=437299]: Done uploading segments... Dec 26 09:44:04.753119550 (498h15m4.624202595s) [upload=437299]: Upload finished. err: <nil> Duration: 53.913790928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXX..X.......X.....XX.XX.XXX.....X..X.....X.X...X..XXXXXXX..] duration: 6.078861201s trace: https://jaeger.dev.storj.io/trace/5961db0f7939a572 FAIL (2025-12-26 09:47:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 6R5AHGWPFRAZW3442VDKUX5ILHSJF7GMT6OTOU75O5E7PFV44K7Q) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: T2E2THTWPZZ2KNYZBRL2RAINUDJK3MOCZHMVWVG7EBPIJCOA5SRQ) context deadline exceeded ======LOGS====== Dec 26 09:46:08.165062995 (498h17m8.03614605s) [upload=437335]: Starting upload Dec 26 09:46:08.165149940 (498h17m8.036232985s) [upload=437335]: Done starting upload Dec 26 09:46:08.165273245 (498h17m8.03635629s) [upload=437335]: Uploading segments... Dec 26 09:46:08.165437147 (498h17m8.036520192s) [upload=437335]: Got next segment. Inline: false Dec 26 09:46:08.165478266 (498h17m8.036561311s) [upload=437335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:46:08.165483325 (498h17m8.03656637s) [upload=437335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:46:08.414578364 (498h17m8.285661399s) [seg_pos={0 0}, upload=437335]: Begin upload segment... Dec 26 09:46:08.414622609 (498h17m8.285705644s) [seg_pos={0 0}, upload=437335]: Done begin upload segment. Dec 26 09:46:08.414648839 (498h17m8.285731884s) [noise=true , address=145.79.227.17:4970 , piece_id=Q6QVMEARDXMYYLHXZVSNZYMPD2J4GQUJT7FZH6X5CZPGRJ2CR2HA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.414740844 (498h17m8.285823879s) [noise=true , address=45.91.116.6:4973 , piece_id=ZGLKRRFLHU6YYEJ2ML23D6LWU5NPQPABC34L5CEHDA66JTASJ5GA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.415427883 (498h17m8.286510918s) [noise=true , address=145.79.227.36:4517 , piece_id=BD3YHDVEDPBNALVN32SLT6DZJPDBOS2TSM5TW7YT72WJUY7ZR7JA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.415816282 (498h17m8.286899317s) [noise=true , address=145.79.227.41:4086 , piece_id=22EPMHWC2W3R5ZUJODJXBEPZVN35NTPCA7N2ZTAMF5OPZXETQP5Q, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.416313959 (498h17m8.287396995s) [noise=true , address=145.79.227.45:4230 , piece_id=67GCDFMYPGATGWD7SEKQVMEB65XARZ3EF7O62SHFVGZGZMAE42JQ, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.416644099 (498h17m8.287727134s) [noise=true , address=145.79.227.29:4941 , piece_id=FHAHQWT3ANQOCQYMQHW7X6QLHGFJLZYACNYGS2OW4NNGJXNF3QFQ, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.416941124 (498h17m8.288024159s) [noise=true , address=145.79.227.2:4115 , piece_id=HTRB5757CCREXIYVDK7MS2T6VIYBATJ6VMMJILH353VN4KELEMTA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.417223171 (498h17m8.288306206s) [noise=true , address=77.90.31.13:4629 , piece_id=6R5AHGWPFRAZW3442VDKUX5ILHSJF7GMT6OTOU75O5E7PFV44K7Q, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.417555835 (498h17m8.28863887s) [noise=true , address=145.79.227.42:4123 , piece_id=JAOPHXKNAJOBIEK4RYA2OWEUOAMP462WJQJJFEWSYCRK62R5CJRA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.418732335 (498h17m8.28981537s) [noise=true , address=45.91.116.5:4300 , piece_id=SMKSHPROTHQ26ZNVUAVIMSX5DU5ZAD5V6TRPARSWZZBCOORTT5XQ, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.418885456 (498h17m8.289968491s) [noise=true , address=45.91.116.11:4120 , piece_id=T2E2THTWPZZ2KNYZBRL2RAINUDJK3MOCZHMVWVG7EBPIJCOA5SRQ, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.419814956 (498h17m8.290898001s) [noise=true , address=45.91.116.14:4190 , piece_id=D7TATQKJASAAGGVPYRW3HH26R4DNQO3H7XNSNO6HGCVVBJTS554Q, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.420438995 (498h17m8.29152204s) [noise=true , address=45.91.116.4:4940 , piece_id=MEW6HWSNGGOPG7ELLDJLOPU7AJJ5Z7ERMNPRA3GCZKXR5RVYU6UA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.421883214 (498h17m8.292966259s) [noise=true , address=145.79.227.46:4603 , piece_id=XXEBZ4TQ2MGMDL3F5KYTZIZVA6HN376MKW5IKSITC2ED5YIMA5EA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.423567993 (498h17m8.294651028s) [noise=true , address=145.79.227.39:4015 , piece_id=YKJBRUPAJLX64MV4YS4WUG2LWOZA42WEF4BOGQYMGMRXRHSZUIKA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.425066296 (498h17m8.296149331s) [noise=true , address=145.79.227.37:4566 , piece_id=YNVC2E7IYVDAKT6QTBLTXV35Q2P73DFDWXCZ4LULV4PJ4XUSKE5Q, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:10.138884602 (498h17m10.009967658s) [upload=437335]: Next returned nil segment Dec 26 09:46:10.138909960 (498h17m10.009992995s) [upload=437335]: Waiting for error group managing segments... Dec 26 09:46:10.413906521 (498h17m10.284989576s) [noise=true , address=145.79.227.37:4566 , piece_id=YNVC2E7IYVDAKT6QTBLTXV35Q2P73DFDWXCZ4LULV4PJ4XUSKE5Q, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.427869810 (498h17m10.298952865s) [noise=true , address=145.79.227.45:4230 , piece_id=67GCDFMYPGATGWD7SEKQVMEB65XARZ3EF7O62SHFVGZGZMAE42JQ, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.430136766 (498h17m10.301219801s) [noise=true , address=145.79.227.29:4941 , piece_id=FHAHQWT3ANQOCQYMQHW7X6QLHGFJLZYACNYGS2OW4NNGJXNF3QFQ, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.430493807 (498h17m10.301576842s) [noise=true , address=145.79.227.41:4086 , piece_id=22EPMHWC2W3R5ZUJODJXBEPZVN35NTPCA7N2ZTAMF5OPZXETQP5Q, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.434724542 (498h17m10.305807587s) [noise=true , address=145.79.227.39:4015 , piece_id=YKJBRUPAJLX64MV4YS4WUG2LWOZA42WEF4BOGQYMGMRXRHSZUIKA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.435101160 (498h17m10.306184195s) [noise=true , address=145.79.227.46:4603 , piece_id=XXEBZ4TQ2MGMDL3F5KYTZIZVA6HN376MKW5IKSITC2ED5YIMA5EA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.443912207 (498h17m10.314995252s) [noise=true , address=145.79.227.17:4970 , piece_id=Q6QVMEARDXMYYLHXZVSNZYMPD2J4GQUJT7FZH6X5CZPGRJ2CR2HA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.444769119 (498h17m10.315852164s) [noise=true , address=145.79.227.42:4123 , piece_id=JAOPHXKNAJOBIEK4RYA2OWEUOAMP462WJQJJFEWSYCRK62R5CJRA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.446579456 (498h17m10.317662502s) [noise=true , address=145.79.227.36:4517 , piece_id=BD3YHDVEDPBNALVN32SLT6DZJPDBOS2TSM5TW7YT72WJUY7ZR7JA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.459713005 (498h17m10.33079605s) [noise=true , address=145.79.227.2:4115 , piece_id=HTRB5757CCREXIYVDK7MS2T6VIYBATJ6VMMJILH353VN4KELEMTA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.668029625 (498h17m10.53911266s) [noise=true , address=45.91.116.11:4120 , piece_id=T2E2THTWPZZ2KNYZBRL2RAINUDJK3MOCZHMVWVG7EBPIJCOA5SRQ, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.732776044 (498h17m10.603859079s) [noise=true , address=77.90.31.13:4629 , piece_id=6R5AHGWPFRAZW3442VDKUX5ILHSJF7GMT6OTOU75O5E7PFV44K7Q, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.765684757 (498h17m10.636767802s) [noise=true , address=45.91.116.6:4973 , piece_id=ZGLKRRFLHU6YYEJ2ML23D6LWU5NPQPABC34L5CEHDA66JTASJ5GA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.765707080 (498h17m10.636790115s) [seg_pos={0 0}, upload=437335]: Segment reached optimal threshold of 13 pieces. Dec 26 09:46:10.766046667 (498h17m10.637129702s) [noise=true , address=45.91.116.14:4190 , piece_id=D7TATQKJASAAGGVPYRW3HH26R4DNQO3H7XNSNO6HGCVVBJTS554Q, seg_pos={0 0}, upload=437335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 26 09:46:10.766054642 (498h17m10.637137677s) [noise=true , address=45.91.116.14:4190 , piece_id=D7TATQKJASAAGGVPYRW3HH26R4DNQO3H7XNSNO6HGCVVBJTS554Q, seg_pos={0 0}, upload=437335]: Long tail cancellation due to optimal threshold Dec 26 09:46:10.766348301 (498h17m10.637431336s) [noise=true , address=45.91.116.5:4300 , piece_id=SMKSHPROTHQ26ZNVUAVIMSX5DU5ZAD5V6TRPARSWZZBCOORTT5XQ, seg_pos={0 0}, upload=437335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:51552->45.91.116.5:4300: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 09:46:10.766354824 (498h17m10.637437869s) [noise=true , address=45.91.116.5:4300 , piece_id=SMKSHPROTHQ26ZNVUAVIMSX5DU5ZAD5V6TRPARSWZZBCOORTT5XQ, seg_pos={0 0}, upload=437335]: Long tail cancellation due to optimal threshold Dec 26 09:46:10.766548141 (498h17m10.637631176s) [noise=true , address=45.91.116.4:4940 , piece_id=MEW6HWSNGGOPG7ELLDJLOPU7AJJ5Z7ERMNPRA3GCZKXR5RVYU6UA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: read tcp 188.166.225.170:53780->45.91.116.4:4940: use of closed network connection; context canceled Dec 26 09:46:10.766557589 (498h17m10.637640634s) [noise=true , address=45.91.116.4:4940 , piece_id=MEW6HWSNGGOPG7ELLDJLOPU7AJJ5Z7ERMNPRA3GCZKXR5RVYU6UA, seg_pos={0 0}, upload=437335]: Long tail cancellation due to optimal threshold Dec 26 09:46:10.766572508 (498h17m10.637655543s) [seg_pos={0 0}, upload=437335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:46:10.766639335 (498h17m10.637722381s) [upload=437335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:46:10.766651428 (498h17m10.637734463s) [upload=437335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:46:11.066893972 (498h17m10.937977057s) [upload=437335]: Done uploading segments... Dec 26 09:46:11.066930682 (498h17m10.938013767s) [upload=437335]: Upload finished. err: <nil> Duration: 6.078900056s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XXX..X.......X.....XX.XX.XXX.....X..X.....X.X...X..XXXXXXX.] duration: 6.413786368s trace: https://jaeger.dev.storj.io/trace/5961db0f7939a572 FAIL (2025-12-26 09:47:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 6R5AHGWPFRAZW3442VDKUX5ILHSJF7GMT6OTOU75O5E7PFV44K7Q) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: T2E2THTWPZZ2KNYZBRL2RAINUDJK3MOCZHMVWVG7EBPIJCOA5SRQ) context deadline exceeded ======LOGS====== Dec 26 09:46:08.165062995 (498h17m8.03614605s) [upload=437335]: Starting upload Dec 26 09:46:08.165149940 (498h17m8.036232985s) [upload=437335]: Done starting upload Dec 26 09:46:08.165273245 (498h17m8.03635629s) [upload=437335]: Uploading segments... Dec 26 09:46:08.165437147 (498h17m8.036520192s) [upload=437335]: Got next segment. Inline: false Dec 26 09:46:08.165478266 (498h17m8.036561311s) [upload=437335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:46:08.165483325 (498h17m8.03656637s) [upload=437335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:46:08.414578364 (498h17m8.285661399s) [seg_pos={0 0}, upload=437335]: Begin upload segment... Dec 26 09:46:08.414622609 (498h17m8.285705644s) [seg_pos={0 0}, upload=437335]: Done begin upload segment. Dec 26 09:46:08.414648839 (498h17m8.285731884s) [noise=true , address=145.79.227.17:4970 , piece_id=Q6QVMEARDXMYYLHXZVSNZYMPD2J4GQUJT7FZH6X5CZPGRJ2CR2HA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.414740844 (498h17m8.285823879s) [noise=true , address=45.91.116.6:4973 , piece_id=ZGLKRRFLHU6YYEJ2ML23D6LWU5NPQPABC34L5CEHDA66JTASJ5GA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.415427883 (498h17m8.286510918s) [noise=true , address=145.79.227.36:4517 , piece_id=BD3YHDVEDPBNALVN32SLT6DZJPDBOS2TSM5TW7YT72WJUY7ZR7JA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.415816282 (498h17m8.286899317s) [noise=true , address=145.79.227.41:4086 , piece_id=22EPMHWC2W3R5ZUJODJXBEPZVN35NTPCA7N2ZTAMF5OPZXETQP5Q, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.416313959 (498h17m8.287396995s) [noise=true , address=145.79.227.45:4230 , piece_id=67GCDFMYPGATGWD7SEKQVMEB65XARZ3EF7O62SHFVGZGZMAE42JQ, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.416644099 (498h17m8.287727134s) [noise=true , address=145.79.227.29:4941 , piece_id=FHAHQWT3ANQOCQYMQHW7X6QLHGFJLZYACNYGS2OW4NNGJXNF3QFQ, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.416941124 (498h17m8.288024159s) [noise=true , address=145.79.227.2:4115 , piece_id=HTRB5757CCREXIYVDK7MS2T6VIYBATJ6VMMJILH353VN4KELEMTA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.417223171 (498h17m8.288306206s) [noise=true , address=77.90.31.13:4629 , piece_id=6R5AHGWPFRAZW3442VDKUX5ILHSJF7GMT6OTOU75O5E7PFV44K7Q, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.417555835 (498h17m8.28863887s) [noise=true , address=145.79.227.42:4123 , piece_id=JAOPHXKNAJOBIEK4RYA2OWEUOAMP462WJQJJFEWSYCRK62R5CJRA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.418732335 (498h17m8.28981537s) [noise=true , address=45.91.116.5:4300 , piece_id=SMKSHPROTHQ26ZNVUAVIMSX5DU5ZAD5V6TRPARSWZZBCOORTT5XQ, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.418885456 (498h17m8.289968491s) [noise=true , address=45.91.116.11:4120 , piece_id=T2E2THTWPZZ2KNYZBRL2RAINUDJK3MOCZHMVWVG7EBPIJCOA5SRQ, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.419814956 (498h17m8.290898001s) [noise=true , address=45.91.116.14:4190 , piece_id=D7TATQKJASAAGGVPYRW3HH26R4DNQO3H7XNSNO6HGCVVBJTS554Q, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.420438995 (498h17m8.29152204s) [noise=true , address=45.91.116.4:4940 , piece_id=MEW6HWSNGGOPG7ELLDJLOPU7AJJ5Z7ERMNPRA3GCZKXR5RVYU6UA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.421883214 (498h17m8.292966259s) [noise=true , address=145.79.227.46:4603 , piece_id=XXEBZ4TQ2MGMDL3F5KYTZIZVA6HN376MKW5IKSITC2ED5YIMA5EA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.423567993 (498h17m8.294651028s) [noise=true , address=145.79.227.39:4015 , piece_id=YKJBRUPAJLX64MV4YS4WUG2LWOZA42WEF4BOGQYMGMRXRHSZUIKA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.425066296 (498h17m8.296149331s) [noise=true , address=145.79.227.37:4566 , piece_id=YNVC2E7IYVDAKT6QTBLTXV35Q2P73DFDWXCZ4LULV4PJ4XUSKE5Q, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:10.138884602 (498h17m10.009967658s) [upload=437335]: Next returned nil segment Dec 26 09:46:10.138909960 (498h17m10.009992995s) [upload=437335]: Waiting for error group managing segments... Dec 26 09:46:10.413906521 (498h17m10.284989576s) [noise=true , address=145.79.227.37:4566 , piece_id=YNVC2E7IYVDAKT6QTBLTXV35Q2P73DFDWXCZ4LULV4PJ4XUSKE5Q, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.427869810 (498h17m10.298952865s) [noise=true , address=145.79.227.45:4230 , piece_id=67GCDFMYPGATGWD7SEKQVMEB65XARZ3EF7O62SHFVGZGZMAE42JQ, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.430136766 (498h17m10.301219801s) [noise=true , address=145.79.227.29:4941 , piece_id=FHAHQWT3ANQOCQYMQHW7X6QLHGFJLZYACNYGS2OW4NNGJXNF3QFQ, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.430493807 (498h17m10.301576842s) [noise=true , address=145.79.227.41:4086 , piece_id=22EPMHWC2W3R5ZUJODJXBEPZVN35NTPCA7N2ZTAMF5OPZXETQP5Q, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.434724542 (498h17m10.305807587s) [noise=true , address=145.79.227.39:4015 , piece_id=YKJBRUPAJLX64MV4YS4WUG2LWOZA42WEF4BOGQYMGMRXRHSZUIKA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.435101160 (498h17m10.306184195s) [noise=true , address=145.79.227.46:4603 , piece_id=XXEBZ4TQ2MGMDL3F5KYTZIZVA6HN376MKW5IKSITC2ED5YIMA5EA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.443912207 (498h17m10.314995252s) [noise=true , address=145.79.227.17:4970 , piece_id=Q6QVMEARDXMYYLHXZVSNZYMPD2J4GQUJT7FZH6X5CZPGRJ2CR2HA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.444769119 (498h17m10.315852164s) [noise=true , address=145.79.227.42:4123 , piece_id=JAOPHXKNAJOBIEK4RYA2OWEUOAMP462WJQJJFEWSYCRK62R5CJRA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.446579456 (498h17m10.317662502s) [noise=true , address=145.79.227.36:4517 , piece_id=BD3YHDVEDPBNALVN32SLT6DZJPDBOS2TSM5TW7YT72WJUY7ZR7JA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.459713005 (498h17m10.33079605s) [noise=true , address=145.79.227.2:4115 , piece_id=HTRB5757CCREXIYVDK7MS2T6VIYBATJ6VMMJILH353VN4KELEMTA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.668029625 (498h17m10.53911266s) [noise=true , address=45.91.116.11:4120 , piece_id=T2E2THTWPZZ2KNYZBRL2RAINUDJK3MOCZHMVWVG7EBPIJCOA5SRQ, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.732776044 (498h17m10.603859079s) [noise=true , address=77.90.31.13:4629 , piece_id=6R5AHGWPFRAZW3442VDKUX5ILHSJF7GMT6OTOU75O5E7PFV44K7Q, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.765684757 (498h17m10.636767802s) [noise=true , address=45.91.116.6:4973 , piece_id=ZGLKRRFLHU6YYEJ2ML23D6LWU5NPQPABC34L5CEHDA66JTASJ5GA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.765707080 (498h17m10.636790115s) [seg_pos={0 0}, upload=437335]: Segment reached optimal threshold of 13 pieces. Dec 26 09:46:10.766046667 (498h17m10.637129702s) [noise=true , address=45.91.116.14:4190 , piece_id=D7TATQKJASAAGGVPYRW3HH26R4DNQO3H7XNSNO6HGCVVBJTS554Q, seg_pos={0 0}, upload=437335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 26 09:46:10.766054642 (498h17m10.637137677s) [noise=true , address=45.91.116.14:4190 , piece_id=D7TATQKJASAAGGVPYRW3HH26R4DNQO3H7XNSNO6HGCVVBJTS554Q, seg_pos={0 0}, upload=437335]: Long tail cancellation due to optimal threshold Dec 26 09:46:10.766348301 (498h17m10.637431336s) [noise=true , address=45.91.116.5:4300 , piece_id=SMKSHPROTHQ26ZNVUAVIMSX5DU5ZAD5V6TRPARSWZZBCOORTT5XQ, seg_pos={0 0}, upload=437335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:51552->45.91.116.5:4300: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 09:46:10.766354824 (498h17m10.637437869s) [noise=true , address=45.91.116.5:4300 , piece_id=SMKSHPROTHQ26ZNVUAVIMSX5DU5ZAD5V6TRPARSWZZBCOORTT5XQ, seg_pos={0 0}, upload=437335]: Long tail cancellation due to optimal threshold Dec 26 09:46:10.766548141 (498h17m10.637631176s) [noise=true , address=45.91.116.4:4940 , piece_id=MEW6HWSNGGOPG7ELLDJLOPU7AJJ5Z7ERMNPRA3GCZKXR5RVYU6UA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: read tcp 188.166.225.170:53780->45.91.116.4:4940: use of closed network connection; context canceled Dec 26 09:46:10.766557589 (498h17m10.637640634s) [noise=true , address=45.91.116.4:4940 , piece_id=MEW6HWSNGGOPG7ELLDJLOPU7AJJ5Z7ERMNPRA3GCZKXR5RVYU6UA, seg_pos={0 0}, upload=437335]: Long tail cancellation due to optimal threshold Dec 26 09:46:10.766572508 (498h17m10.637655543s) [seg_pos={0 0}, upload=437335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:46:10.766639335 (498h17m10.637722381s) [upload=437335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:46:10.766651428 (498h17m10.637734463s) [upload=437335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:46:11.066893972 (498h17m10.937977057s) [upload=437335]: Done uploading segments... Dec 26 09:46:11.066930682 (498h17m10.938013767s) [upload=437335]: Upload finished. err: <nil> Duration: 6.413802808s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XXX..X.......X.....XX.XX.XXX.....X..X.....X.X...X..XXXXXXX] duration: 30.003785286s trace: https://jaeger.dev.storj.io/trace/5961db0f7939a572 FAIL (2025-12-26 09:46:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 6R5AHGWPFRAZW3442VDKUX5ILHSJF7GMT6OTOU75O5E7PFV44K7Q) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: T2E2THTWPZZ2KNYZBRL2RAINUDJK3MOCZHMVWVG7EBPIJCOA5SRQ) context deadline exceeded ======LOGS====== Dec 26 09:46:08.165062995 (498h17m8.03614605s) [upload=437335]: Starting upload Dec 26 09:46:08.165149940 (498h17m8.036232985s) [upload=437335]: Done starting upload Dec 26 09:46:08.165273245 (498h17m8.03635629s) [upload=437335]: Uploading segments... Dec 26 09:46:08.165437147 (498h17m8.036520192s) [upload=437335]: Got next segment. Inline: false Dec 26 09:46:08.165478266 (498h17m8.036561311s) [upload=437335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:46:08.165483325 (498h17m8.03656637s) [upload=437335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:46:08.414578364 (498h17m8.285661399s) [seg_pos={0 0}, upload=437335]: Begin upload segment... Dec 26 09:46:08.414622609 (498h17m8.285705644s) [seg_pos={0 0}, upload=437335]: Done begin upload segment. Dec 26 09:46:08.414648839 (498h17m8.285731884s) [noise=true , address=145.79.227.17:4970 , piece_id=Q6QVMEARDXMYYLHXZVSNZYMPD2J4GQUJT7FZH6X5CZPGRJ2CR2HA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.414740844 (498h17m8.285823879s) [noise=true , address=45.91.116.6:4973 , piece_id=ZGLKRRFLHU6YYEJ2ML23D6LWU5NPQPABC34L5CEHDA66JTASJ5GA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.415427883 (498h17m8.286510918s) [noise=true , address=145.79.227.36:4517 , piece_id=BD3YHDVEDPBNALVN32SLT6DZJPDBOS2TSM5TW7YT72WJUY7ZR7JA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.415816282 (498h17m8.286899317s) [noise=true , address=145.79.227.41:4086 , piece_id=22EPMHWC2W3R5ZUJODJXBEPZVN35NTPCA7N2ZTAMF5OPZXETQP5Q, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.416313959 (498h17m8.287396995s) [noise=true , address=145.79.227.45:4230 , piece_id=67GCDFMYPGATGWD7SEKQVMEB65XARZ3EF7O62SHFVGZGZMAE42JQ, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.416644099 (498h17m8.287727134s) [noise=true , address=145.79.227.29:4941 , piece_id=FHAHQWT3ANQOCQYMQHW7X6QLHGFJLZYACNYGS2OW4NNGJXNF3QFQ, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.416941124 (498h17m8.288024159s) [noise=true , address=145.79.227.2:4115 , piece_id=HTRB5757CCREXIYVDK7MS2T6VIYBATJ6VMMJILH353VN4KELEMTA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.417223171 (498h17m8.288306206s) [noise=true , address=77.90.31.13:4629 , piece_id=6R5AHGWPFRAZW3442VDKUX5ILHSJF7GMT6OTOU75O5E7PFV44K7Q, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.417555835 (498h17m8.28863887s) [noise=true , address=145.79.227.42:4123 , piece_id=JAOPHXKNAJOBIEK4RYA2OWEUOAMP462WJQJJFEWSYCRK62R5CJRA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.418732335 (498h17m8.28981537s) [noise=true , address=45.91.116.5:4300 , piece_id=SMKSHPROTHQ26ZNVUAVIMSX5DU5ZAD5V6TRPARSWZZBCOORTT5XQ, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.418885456 (498h17m8.289968491s) [noise=true , address=45.91.116.11:4120 , piece_id=T2E2THTWPZZ2KNYZBRL2RAINUDJK3MOCZHMVWVG7EBPIJCOA5SRQ, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.419814956 (498h17m8.290898001s) [noise=true , address=45.91.116.14:4190 , piece_id=D7TATQKJASAAGGVPYRW3HH26R4DNQO3H7XNSNO6HGCVVBJTS554Q, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.420438995 (498h17m8.29152204s) [noise=true , address=45.91.116.4:4940 , piece_id=MEW6HWSNGGOPG7ELLDJLOPU7AJJ5Z7ERMNPRA3GCZKXR5RVYU6UA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.421883214 (498h17m8.292966259s) [noise=true , address=145.79.227.46:4603 , piece_id=XXEBZ4TQ2MGMDL3F5KYTZIZVA6HN376MKW5IKSITC2ED5YIMA5EA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.423567993 (498h17m8.294651028s) [noise=true , address=145.79.227.39:4015 , piece_id=YKJBRUPAJLX64MV4YS4WUG2LWOZA42WEF4BOGQYMGMRXRHSZUIKA, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:08.425066296 (498h17m8.296149331s) [noise=true , address=145.79.227.37:4566 , piece_id=YNVC2E7IYVDAKT6QTBLTXV35Q2P73DFDWXCZ4LULV4PJ4XUSKE5Q, seg_pos={0 0}, upload=437335]: Uploading piece... Dec 26 09:46:10.138884602 (498h17m10.009967658s) [upload=437335]: Next returned nil segment Dec 26 09:46:10.138909960 (498h17m10.009992995s) [upload=437335]: Waiting for error group managing segments... Dec 26 09:46:10.413906521 (498h17m10.284989576s) [noise=true , address=145.79.227.37:4566 , piece_id=YNVC2E7IYVDAKT6QTBLTXV35Q2P73DFDWXCZ4LULV4PJ4XUSKE5Q, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.427869810 (498h17m10.298952865s) [noise=true , address=145.79.227.45:4230 , piece_id=67GCDFMYPGATGWD7SEKQVMEB65XARZ3EF7O62SHFVGZGZMAE42JQ, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.430136766 (498h17m10.301219801s) [noise=true , address=145.79.227.29:4941 , piece_id=FHAHQWT3ANQOCQYMQHW7X6QLHGFJLZYACNYGS2OW4NNGJXNF3QFQ, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.430493807 (498h17m10.301576842s) [noise=true , address=145.79.227.41:4086 , piece_id=22EPMHWC2W3R5ZUJODJXBEPZVN35NTPCA7N2ZTAMF5OPZXETQP5Q, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.434724542 (498h17m10.305807587s) [noise=true , address=145.79.227.39:4015 , piece_id=YKJBRUPAJLX64MV4YS4WUG2LWOZA42WEF4BOGQYMGMRXRHSZUIKA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.435101160 (498h17m10.306184195s) [noise=true , address=145.79.227.46:4603 , piece_id=XXEBZ4TQ2MGMDL3F5KYTZIZVA6HN376MKW5IKSITC2ED5YIMA5EA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.443912207 (498h17m10.314995252s) [noise=true , address=145.79.227.17:4970 , piece_id=Q6QVMEARDXMYYLHXZVSNZYMPD2J4GQUJT7FZH6X5CZPGRJ2CR2HA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.444769119 (498h17m10.315852164s) [noise=true , address=145.79.227.42:4123 , piece_id=JAOPHXKNAJOBIEK4RYA2OWEUOAMP462WJQJJFEWSYCRK62R5CJRA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.446579456 (498h17m10.317662502s) [noise=true , address=145.79.227.36:4517 , piece_id=BD3YHDVEDPBNALVN32SLT6DZJPDBOS2TSM5TW7YT72WJUY7ZR7JA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.459713005 (498h17m10.33079605s) [noise=true , address=145.79.227.2:4115 , piece_id=HTRB5757CCREXIYVDK7MS2T6VIYBATJ6VMMJILH353VN4KELEMTA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.668029625 (498h17m10.53911266s) [noise=true , address=45.91.116.11:4120 , piece_id=T2E2THTWPZZ2KNYZBRL2RAINUDJK3MOCZHMVWVG7EBPIJCOA5SRQ, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.732776044 (498h17m10.603859079s) [noise=true , address=77.90.31.13:4629 , piece_id=6R5AHGWPFRAZW3442VDKUX5ILHSJF7GMT6OTOU75O5E7PFV44K7Q, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.765684757 (498h17m10.636767802s) [noise=true , address=45.91.116.6:4973 , piece_id=ZGLKRRFLHU6YYEJ2ML23D6LWU5NPQPABC34L5CEHDA66JTASJ5GA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: <nil> Dec 26 09:46:10.765707080 (498h17m10.636790115s) [seg_pos={0 0}, upload=437335]: Segment reached optimal threshold of 13 pieces. Dec 26 09:46:10.766046667 (498h17m10.637129702s) [noise=true , address=45.91.116.14:4190 , piece_id=D7TATQKJASAAGGVPYRW3HH26R4DNQO3H7XNSNO6HGCVVBJTS554Q, seg_pos={0 0}, upload=437335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 26 09:46:10.766054642 (498h17m10.637137677s) [noise=true , address=45.91.116.14:4190 , piece_id=D7TATQKJASAAGGVPYRW3HH26R4DNQO3H7XNSNO6HGCVVBJTS554Q, seg_pos={0 0}, upload=437335]: Long tail cancellation due to optimal threshold Dec 26 09:46:10.766348301 (498h17m10.637431336s) [noise=true , address=45.91.116.5:4300 , piece_id=SMKSHPROTHQ26ZNVUAVIMSX5DU5ZAD5V6TRPARSWZZBCOORTT5XQ, seg_pos={0 0}, upload=437335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:51552->45.91.116.5:4300: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 09:46:10.766354824 (498h17m10.637437869s) [noise=true , address=45.91.116.5:4300 , piece_id=SMKSHPROTHQ26ZNVUAVIMSX5DU5ZAD5V6TRPARSWZZBCOORTT5XQ, seg_pos={0 0}, upload=437335]: Long tail cancellation due to optimal threshold Dec 26 09:46:10.766548141 (498h17m10.637631176s) [noise=true , address=45.91.116.4:4940 , piece_id=MEW6HWSNGGOPG7ELLDJLOPU7AJJ5Z7ERMNPRA3GCZKXR5RVYU6UA, seg_pos={0 0}, upload=437335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: read tcp 188.166.225.170:53780->45.91.116.4:4940: use of closed network connection; context canceled Dec 26 09:46:10.766557589 (498h17m10.637640634s) [noise=true , address=45.91.116.4:4940 , piece_id=MEW6HWSNGGOPG7ELLDJLOPU7AJJ5Z7ERMNPRA3GCZKXR5RVYU6UA, seg_pos={0 0}, upload=437335]: Long tail cancellation due to optimal threshold Dec 26 09:46:10.766572508 (498h17m10.637655543s) [seg_pos={0 0}, upload=437335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:46:10.766639335 (498h17m10.637722381s) [upload=437335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:46:10.766651428 (498h17m10.637734463s) [upload=437335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:46:11.066893972 (498h17m10.937977057s) [upload=437335]: Done uploading segments... Dec 26 09:46:11.066930682 (498h17m10.938013767s) [upload=437335]: Upload finished. err: <nil> Duration: 30.003818148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X......X...] duration: 28.06524006s trace: https://jaeger.dev.storj.io/trace/2340efa351101f24 FAIL (2025-12-26 09:46:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 26 09:44:00.880088265 (498h15m0.751171301s) [noise=true , address=45.91.116.7:4016 , piece_id=JAZZQAOSDSXEAJS57ZNGRUE4RLXQQ6MPBDDJGPD4QNSLU56TZYSQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.880229845 (498h15m0.75131288s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.880757570 (498h15m0.751840615s) [noise=true , address=145.79.227.30:4765 , piece_id=IJRWZ3YBUSGZJ3WJKXEEOWWIFMO25CMBYR2PO537EKHTQHB3XP2Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.881224599 (498h15m0.752307644s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.882867498 (498h15m0.753950533s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.883629057 (498h15m0.754712102s) [noise=true , address=45.91.116.7:4042 , piece_id=QOYKGPE2E7SSX7YCPVHMZYVXOH43MW5ODEJMYEACSUDLPJLP6QHA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.884590658 (498h15m0.755673734s) [noise=true , address=45.91.116.9:4272 , piece_id=4LG7DWD5GJEXX4KGRS6QROVWBXGCUZ4LWOBMIPI2AEK2Q3NIK7XQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.884836046 (498h15m0.755919081s) [noise=true , address=45.91.116.10:4104 , piece_id=5YHM2XVPLW5BUDOFKSRPAOU6FAQEZUFVJQU2PKAXMLSGFBC3L73Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.886094181 (498h15m0.757177236s) [noise=true , address=145.79.227.7:4322 , piece_id=4IXKAMY6AB6HRGNMZLLMZ2WWUUHXRGZGD35VZMORUVG4UHV2Q4PA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.886114190 (498h15m0.757197235s) [noise=true , address=45.91.116.4:4944 , piece_id=PLIUZEYKODIE3JKUFVI43FZ47DYVAE6CGFXT45N6MBOLYJV626UQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.887478658 (498h15m0.758561693s) [noise=true , address=77.90.31.12:4598 , piece_id=L33QFG7QEALMVTS2IOF7DZOA4LCFMKKRPG4E77EALQK2NG622Q2Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.887528633 (498h15m0.758611668s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.888256428 (498h15m0.759339474s) [noise=true , address=45.91.116.14:4195 , piece_id=QEOVODBM24X6ROM2ME3GCTPVUHFHDE5JBWAQSECCRITOAJIFR4SA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.888915033 (498h15m0.759998078s) [noise=true , address=145.79.227.44:4203 , piece_id=6XBUBLM3UGODNA2INBVZLSKVQSQIVKECAYEFMHULVHI4FN7LHMKQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.890072357 (498h15m0.761155392s) [noise=true , address=45.91.116.11:4129 , piece_id=QCGCBLS7PHXGRAB2ZGPUDKL5ADSDMCZHHSJPQVOSPHXMX7O6UIJA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.890947614 (498h15m0.762030659s) [noise=true , address=77.90.31.14:4675 , piece_id=BHLZL54ETYKAACJYLGD3JGOJEIM7IVOS4BIIKDN3ZHDEDBVQBMQA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:02.349863155 (498h15m2.22094619s) [upload=437299]: Next returned nil segment Dec 26 09:44:02.349900766 (498h15m2.220983811s) [upload=437299]: Waiting for error group managing segments... Dec 26 09:44:02.781836534 (498h15m2.65291958s) [noise=true , address=145.79.227.7:4322 , piece_id=4IXKAMY6AB6HRGNMZLLMZ2WWUUHXRGZGD35VZMORUVG4UHV2Q4PA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:02.928532198 (498h15m2.799615243s) [noise=true , address=145.79.227.34:4937 , piece_id=RRTHSCYGQXVBGWD4IATI7XEBM32RDPVQR7T4J3XJQXUIGWU45GCA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:02.959038798 (498h15m2.830121853s) [noise=true , address=77.90.31.14:4675 , piece_id=BHLZL54ETYKAACJYLGD3JGOJEIM7IVOS4BIIKDN3ZHDEDBVQBMQA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.052394553 (498h15m2.923477598s) [noise=true , address=77.90.31.12:4598 , piece_id=L33QFG7QEALMVTS2IOF7DZOA4LCFMKKRPG4E77EALQK2NG622Q2Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.066899663 (498h15m2.937982708s) [noise=true , address=45.91.116.4:4944 , piece_id=PLIUZEYKODIE3JKUFVI43FZ47DYVAE6CGFXT45N6MBOLYJV626UQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.081280326 (498h15m2.952363381s) [noise=true , address=45.91.116.10:4104 , piece_id=5YHM2XVPLW5BUDOFKSRPAOU6FAQEZUFVJQU2PKAXMLSGFBC3L73Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.096434834 (498h15m2.967517869s) [noise=true , address=45.91.116.7:4016 , piece_id=JAZZQAOSDSXEAJS57ZNGRUE4RLXQQ6MPBDDJGPD4QNSLU56TZYSQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.107137242 (498h15m2.978220287s) [noise=true , address=45.91.116.14:4195 , piece_id=QEOVODBM24X6ROM2ME3GCTPVUHFHDE5JBWAQSECCRITOAJIFR4SA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.176790474 (498h15m3.047873519s) [noise=true , address=145.79.227.40:4074 , piece_id=SPEIB4FDHK7I45NIXB3CLX5HNVNKSQZ5APXE4ULGW3MVPEEEKCOA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.431315263 (498h15m3.302398308s) [noise=true , address=45.91.116.9:4272 , piece_id=4LG7DWD5GJEXX4KGRS6QROVWBXGCUZ4LWOBMIPI2AEK2Q3NIK7XQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.710113361 (498h15m3.581196406s) [noise=true , address=145.79.227.44:4203 , piece_id=6XBUBLM3UGODNA2INBVZLSKVQSQIVKECAYEFMHULVHI4FN7LHMKQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.716995455 (498h15m3.5880785s) [noise=true , address=145.79.227.30:4765 , piece_id=IJRWZ3YBUSGZJ3WJKXEEOWWIFMO25CMBYR2PO537EKHTQHB3XP2Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.183858151 (498h15m4.054941206s) [noise=true , address=45.91.116.11:4129 , piece_id=QCGCBLS7PHXGRAB2ZGPUDKL5ADSDMCZHHSJPQVOSPHXMX7O6UIJA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.462428025 (498h15m4.33351106s) [noise=true , address=45.91.116.7:4042 , piece_id=QOYKGPE2E7SSX7YCPVHMZYVXOH43MW5ODEJMYEACSUDLPJLP6QHA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.462455627 (498h15m4.333538662s) [seg_pos={0 0}, upload=437299]: Segment reached optimal threshold of 14 pieces. Dec 26 09:44:04.462751150 (498h15m4.333834195s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection Dec 26 09:44:04.462761229 (498h15m4.333844284s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463065669 (498h15m4.334148724s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463124842 (498h15m4.334207887s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463174215 (498h15m4.334257361s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FNXVNAv3KeiRqTnsvQkMobyRcNFmPAAjVqSNvgs6MPJH4smVU): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463187411 (498h15m4.334270456s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463448007 (498h15m4.334531052s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463458536 (498h15m4.334541582s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463503002 (498h15m4.334586047s) [seg_pos={0 0}, upload=437299]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 09:44:04.463624553 (498h15m4.334707598s) [upload=437299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:44:04.463631466 (498h15m4.334714521s) [upload=437299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:44:04.753033747 (498h15m4.624116782s) [upload=437299]: Done uploading segments... Dec 26 09:44:04.753119550 (498h15m4.624202595s) [upload=437299]: Upload finished. err: <nil> Duration: 28.065258918s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X......X..] duration: 35.247497751s trace: https://jaeger.dev.storj.io/trace/2340efa351101f24 FAIL (2025-12-26 09:46:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 26 09:44:00.880088265 (498h15m0.751171301s) [noise=true , address=45.91.116.7:4016 , piece_id=JAZZQAOSDSXEAJS57ZNGRUE4RLXQQ6MPBDDJGPD4QNSLU56TZYSQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.880229845 (498h15m0.75131288s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.880757570 (498h15m0.751840615s) [noise=true , address=145.79.227.30:4765 , piece_id=IJRWZ3YBUSGZJ3WJKXEEOWWIFMO25CMBYR2PO537EKHTQHB3XP2Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.881224599 (498h15m0.752307644s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.882867498 (498h15m0.753950533s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.883629057 (498h15m0.754712102s) [noise=true , address=45.91.116.7:4042 , piece_id=QOYKGPE2E7SSX7YCPVHMZYVXOH43MW5ODEJMYEACSUDLPJLP6QHA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.884590658 (498h15m0.755673734s) [noise=true , address=45.91.116.9:4272 , piece_id=4LG7DWD5GJEXX4KGRS6QROVWBXGCUZ4LWOBMIPI2AEK2Q3NIK7XQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.884836046 (498h15m0.755919081s) [noise=true , address=45.91.116.10:4104 , piece_id=5YHM2XVPLW5BUDOFKSRPAOU6FAQEZUFVJQU2PKAXMLSGFBC3L73Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.886094181 (498h15m0.757177236s) [noise=true , address=145.79.227.7:4322 , piece_id=4IXKAMY6AB6HRGNMZLLMZ2WWUUHXRGZGD35VZMORUVG4UHV2Q4PA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.886114190 (498h15m0.757197235s) [noise=true , address=45.91.116.4:4944 , piece_id=PLIUZEYKODIE3JKUFVI43FZ47DYVAE6CGFXT45N6MBOLYJV626UQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.887478658 (498h15m0.758561693s) [noise=true , address=77.90.31.12:4598 , piece_id=L33QFG7QEALMVTS2IOF7DZOA4LCFMKKRPG4E77EALQK2NG622Q2Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.887528633 (498h15m0.758611668s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.888256428 (498h15m0.759339474s) [noise=true , address=45.91.116.14:4195 , piece_id=QEOVODBM24X6ROM2ME3GCTPVUHFHDE5JBWAQSECCRITOAJIFR4SA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.888915033 (498h15m0.759998078s) [noise=true , address=145.79.227.44:4203 , piece_id=6XBUBLM3UGODNA2INBVZLSKVQSQIVKECAYEFMHULVHI4FN7LHMKQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.890072357 (498h15m0.761155392s) [noise=true , address=45.91.116.11:4129 , piece_id=QCGCBLS7PHXGRAB2ZGPUDKL5ADSDMCZHHSJPQVOSPHXMX7O6UIJA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.890947614 (498h15m0.762030659s) [noise=true , address=77.90.31.14:4675 , piece_id=BHLZL54ETYKAACJYLGD3JGOJEIM7IVOS4BIIKDN3ZHDEDBVQBMQA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:02.349863155 (498h15m2.22094619s) [upload=437299]: Next returned nil segment Dec 26 09:44:02.349900766 (498h15m2.220983811s) [upload=437299]: Waiting for error group managing segments... Dec 26 09:44:02.781836534 (498h15m2.65291958s) [noise=true , address=145.79.227.7:4322 , piece_id=4IXKAMY6AB6HRGNMZLLMZ2WWUUHXRGZGD35VZMORUVG4UHV2Q4PA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:02.928532198 (498h15m2.799615243s) [noise=true , address=145.79.227.34:4937 , piece_id=RRTHSCYGQXVBGWD4IATI7XEBM32RDPVQR7T4J3XJQXUIGWU45GCA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:02.959038798 (498h15m2.830121853s) [noise=true , address=77.90.31.14:4675 , piece_id=BHLZL54ETYKAACJYLGD3JGOJEIM7IVOS4BIIKDN3ZHDEDBVQBMQA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.052394553 (498h15m2.923477598s) [noise=true , address=77.90.31.12:4598 , piece_id=L33QFG7QEALMVTS2IOF7DZOA4LCFMKKRPG4E77EALQK2NG622Q2Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.066899663 (498h15m2.937982708s) [noise=true , address=45.91.116.4:4944 , piece_id=PLIUZEYKODIE3JKUFVI43FZ47DYVAE6CGFXT45N6MBOLYJV626UQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.081280326 (498h15m2.952363381s) [noise=true , address=45.91.116.10:4104 , piece_id=5YHM2XVPLW5BUDOFKSRPAOU6FAQEZUFVJQU2PKAXMLSGFBC3L73Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.096434834 (498h15m2.967517869s) [noise=true , address=45.91.116.7:4016 , piece_id=JAZZQAOSDSXEAJS57ZNGRUE4RLXQQ6MPBDDJGPD4QNSLU56TZYSQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.107137242 (498h15m2.978220287s) [noise=true , address=45.91.116.14:4195 , piece_id=QEOVODBM24X6ROM2ME3GCTPVUHFHDE5JBWAQSECCRITOAJIFR4SA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.176790474 (498h15m3.047873519s) [noise=true , address=145.79.227.40:4074 , piece_id=SPEIB4FDHK7I45NIXB3CLX5HNVNKSQZ5APXE4ULGW3MVPEEEKCOA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.431315263 (498h15m3.302398308s) [noise=true , address=45.91.116.9:4272 , piece_id=4LG7DWD5GJEXX4KGRS6QROVWBXGCUZ4LWOBMIPI2AEK2Q3NIK7XQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.710113361 (498h15m3.581196406s) [noise=true , address=145.79.227.44:4203 , piece_id=6XBUBLM3UGODNA2INBVZLSKVQSQIVKECAYEFMHULVHI4FN7LHMKQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.716995455 (498h15m3.5880785s) [noise=true , address=145.79.227.30:4765 , piece_id=IJRWZ3YBUSGZJ3WJKXEEOWWIFMO25CMBYR2PO537EKHTQHB3XP2Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.183858151 (498h15m4.054941206s) [noise=true , address=45.91.116.11:4129 , piece_id=QCGCBLS7PHXGRAB2ZGPUDKL5ADSDMCZHHSJPQVOSPHXMX7O6UIJA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.462428025 (498h15m4.33351106s) [noise=true , address=45.91.116.7:4042 , piece_id=QOYKGPE2E7SSX7YCPVHMZYVXOH43MW5ODEJMYEACSUDLPJLP6QHA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.462455627 (498h15m4.333538662s) [seg_pos={0 0}, upload=437299]: Segment reached optimal threshold of 14 pieces. Dec 26 09:44:04.462751150 (498h15m4.333834195s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection Dec 26 09:44:04.462761229 (498h15m4.333844284s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463065669 (498h15m4.334148724s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463124842 (498h15m4.334207887s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463174215 (498h15m4.334257361s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FNXVNAv3KeiRqTnsvQkMobyRcNFmPAAjVqSNvgs6MPJH4smVU): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463187411 (498h15m4.334270456s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463448007 (498h15m4.334531052s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463458536 (498h15m4.334541582s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463503002 (498h15m4.334586047s) [seg_pos={0 0}, upload=437299]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 09:44:04.463624553 (498h15m4.334707598s) [upload=437299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:44:04.463631466 (498h15m4.334714521s) [upload=437299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:44:04.753033747 (498h15m4.624116782s) [upload=437299]: Done uploading segments... Dec 26 09:44:04.753119550 (498h15m4.624202595s) [upload=437299]: Upload finished. err: <nil> Duration: 35.247519001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...XXX..X.......X.....XX.XX.XXX.....X..X.....X.X...X..XXXXXX] duration: 30.003205253s trace: https://jaeger.dev.storj.io/trace/16a5fa3b1e23b5a1 FAIL (2025-12-26 09:45:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZJ26P7JTMBHT2ZDXUOYJJHRONBY4MUDWJ27LQCXYMDJNEC3LRCPA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: LHTP2PP6OHASBVBOSRZ3HFIMLCESWRER2GYYTV5UEASLDWMEFL2Q) context deadline exceeded ======LOGS====== Dec 26 09:45:27.033587826 (498h16m26.904670861s) [upload=437322]: Starting upload Dec 26 09:45:27.033665323 (498h16m26.904748358s) [upload=437322]: Done starting upload Dec 26 09:45:27.033775893 (498h16m26.904858928s) [upload=437322]: Uploading segments... Dec 26 09:45:27.033915449 (498h16m26.904998494s) [upload=437322]: Got next segment. Inline: false Dec 26 09:45:27.033956527 (498h16m26.905039562s) [upload=437322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:45:27.033961907 (498h16m26.905044942s) [upload=437322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:45:27.279117306 (498h16m27.150200361s) [seg_pos={0 0}, upload=437322]: Begin upload segment... Dec 26 09:45:27.279169154 (498h16m27.150252189s) [seg_pos={0 0}, upload=437322]: Done begin upload segment. Dec 26 09:45:27.279200364 (498h16m27.150283399s) [noise=true , address=145.79.227.7:4293 , piece_id=6SL6R7AC5K2TQ6EHNHXYLAGEPYHMJPUQAU6KPK5A73ZY2V675HCA, seg_pos={0 0}, upload=437322]: Uploading piece... Dec 26 09:45:27.279304672 (498h16m27.150387717s) [noise=true , address=45.91.116.5:4299 , piece_id=FAGTZPDEKWLYH35GBJQSMDSCP5TMGQPSPEPGOBNCE73GXDPPF5BA, seg_pos={0 0}, upload=437322]: Uploading piece... Dec 26 09:45:27.279825474 (498h16m27.150908509s) [noise=true , address=145.79.227.4:4185 , piece_id=3DCXULV7NPQWBDBXHXE6KUJSVJEOCO2XIPLPB5OJMXNKFHI6SIZA, seg_pos={0 0}, upload=437322]: Uploading piece... Dec 26 09:45:27.280154401 (498h16m27.151237436s) [noise=true , address=77.90.31.12:4590 , piece_id=EZRVDMSFTVYFFFAB3RW6T45HSMJJCNT3WQAPPWZJMXNO2UNIAHVQ, seg_pos={0 0}, upload=437322]: Uploading piece... Dec 26 09:45:27.280543943 (498h16m27.151626988s) [noise=true , address=145.79.227.46:4602 , piece_id=YWZTMFX46ISWVOIHXAT7RCNSNIP55WYJZD5FNZXBDCTQKSR5JIWQ, seg_pos={0 0}, upload=437322]: Uploading piece... Dec 26 09:45:27.280883359 (498h16m27.151966394s) [noise=true , address=45.91.116.11:4120 , piece_id=ZJ26P7JTMBHT2ZDXUOYJJHRONBY4MUDWJ27LQCXYMDJNEC3LRCPA, seg_pos={0 0}, upload=437322]: Uploading piece... Dec 26 09:45:27.281162901 (498h16m27.152245936s) [noise=true , address=77.90.31.11:4553 , piece_id=JWSC5SFICPDDIZ2BOEY3EODHXJW6XADBO5V6AZNLG5EPNLHGZSJQ, seg_pos={0 0}, upload=437322]: Uploading piece... Dec 26 09:45:27.281437705 (498h16m27.15252075s) [noise=true , address=145.79.227.33:4871 , piece_id=KWYBG6QDAABJKBBRUUSJA3YCRTUE2PIK6LYQ5BK7SXUDV2BESINQ, seg_pos={0 0}, upload=437322]: Uploading piece... Dec 26 09:45:27.281861391 (498h16m27.152944426s) [noise=true , address=145.79.227.40:4051 , piece_id=N3UI3MBJA6IMYOKXWYQHUPOYADR2V2CGUNTU5PGVBUHVKN4TS6MQ, seg_pos={0 0}, upload=437322]: Uploading piece... Dec 26 09:45:27.281872662 (498h16m27.152955708s) [noise=true , address=45.91.116.13:4155 , piece_id=LHTP2PP6OHASBVBOSRZ3HFIMLCESWRER2GYYTV5UEASLDWMEFL2Q, seg_pos={0 0}, upload=437322]: Uploading piece... Dec 26 09:45:27.282454150 (498h16m27.153537185s) [noise=true , address=145.79.227.48:4302 , piece_id=E43JOBH33LKMOT7ZTXSLTWNNVQDITSFV4GFJC64HMOKSDA4J2B3Q, seg_pos={0 0}, upload=437322]: Uploading piece... Dec 26 09:45:27.282874040 (498h16m27.153957075s) [noise=true , address=77.90.31.18:4808 , piece_id=J23C4DSZQJPWZMP5ZTH3MYA3H3CDUI5QJX7EPDUPWXRNGWYIK2SA, seg_pos={0 0}, upload=437322]: Uploading piece... Dec 26 09:45:27.283278259 (498h16m27.154361304s) [noise=true , address=145.79.227.15:4873 , piece_id=IR6AEM5IDPOAVULTPD6SHFYJSDWQHCCQB2LDAZBIO7QHUJBIKVSQ, seg_pos={0 0}, upload=437322]: Uploading piece... Dec 26 09:45:27.283519428 (498h16m27.154602463s) [noise=true , address=45.91.116.10:4084 , piece_id=WXKRFZKQ7HXCCL3CKU6V2C3O4BUXGE6TZ3B3BQNTQRPBPFXFZ2TA, seg_pos={0 0}, upload=437322]: Uploading piece... Dec 26 09:45:27.283683851 (498h16m27.154766886s) [noise=true , address=145.79.227.6:4259 , piece_id=AABEDNUR6UOLDDXWA2QXYRJNH4JD6SPTQLF5ECTYOOUEA6I4PJQA, seg_pos={0 0}, upload=437322]: Uploading piece... Dec 26 09:45:27.284098080 (498h16m27.155181125s) [noise=true , address=145.79.227.13:4804 , piece_id=C2OGXHOGTRBORM3X5S6ZEBVISXCLU2JGSVFIFJ4LLZ3YAG47XEAQ, seg_pos={0 0}, upload=437322]: Uploading piece... Dec 26 09:45:28.428072214 (498h16m28.299155249s) [noise=true , address=77.90.31.11:4553 , piece_id=JWSC5SFICPDDIZ2BOEY3EODHXJW6XADBO5V6AZNLG5EPNLHGZSJQ, seg_pos={0 0}, upload=437322]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 09:45:28.996520147 (498h16m28.867603192s) [upload=437322]: Next returned nil segment Dec 26 09:45:28.996532090 (498h16m28.867615125s) [upload=437322]: Waiting for error group managing segments... Dec 26 09:45:29.271236194 (498h16m29.142319239s) [noise=true , address=145.79.227.33:4871 , piece_id=KWYBG6QDAABJKBBRUUSJA3YCRTUE2PIK6LYQ5BK7SXUDV2BESINQ, seg_pos={0 0}, upload=437322]: Done uploading piece. err: <nil> Dec 26 09:45:29.281394638 (498h16m29.152477673s) [noise=true , address=145.79.227.46:4602 , piece_id=YWZTMFX46ISWVOIHXAT7RCNSNIP55WYJZD5FNZXBDCTQKSR5JIWQ, seg_pos={0 0}, upload=437322]: Done uploading piece. err: <nil> Dec 26 09:45:29.282401265 (498h16m29.1534843s) [noise=true , address=145.79.227.48:4302 , piece_id=E43JOBH33LKMOT7ZTXSLTWNNVQDITSFV4GFJC64HMOKSDA4J2B3Q, seg_pos={0 0}, upload=437322]: Done uploading piece. err: <nil> Dec 26 09:45:29.288990321 (498h16m29.160073356s) [noise=true , address=145.79.227.4:4185 , piece_id=3DCXULV7NPQWBDBXHXE6KUJSVJEOCO2XIPLPB5OJMXNKFHI6SIZA, seg_pos={0 0}, upload=437322]: Done uploading piece. err: <nil> Dec 26 09:45:29.292156921 (498h16m29.163239966s) [noise=true , address=145.79.227.13:4804 , piece_id=C2OGXHOGTRBORM3X5S6ZEBVISXCLU2JGSVFIFJ4LLZ3YAG47XEAQ, seg_pos={0 0}, upload=437322]: Done uploading piece. err: <nil> Dec 26 09:45:29.308735749 (498h16m29.179818795s) [noise=true , address=145.79.227.7:4293 , piece_id=6SL6R7AC5K2TQ6EHNHXYLAGEPYHMJPUQAU6KPK5A73ZY2V675HCA, seg_pos={0 0}, upload=437322]: Done uploading piece. err: <nil> Dec 26 09:45:29.311809111 (498h16m29.182892156s) [noise=true , address=145.79.227.40:4051 , piece_id=N3UI3MBJA6IMYOKXWYQHUPOYADR2V2CGUNTU5PGVBUHVKN4TS6MQ, seg_pos={0 0}, upload=437322]: Done uploading piece. err: <nil> Dec 26 09:45:29.318695264 (498h16m29.189778299s) [noise=true , address=145.79.227.15:4873 , piece_id=IR6AEM5IDPOAVULTPD6SHFYJSDWQHCCQB2LDAZBIO7QHUJBIKVSQ, seg_pos={0 0}, upload=437322]: Done uploading piece. err: <nil> Dec 26 09:45:29.325957142 (498h16m29.197040177s) [noise=true , address=145.79.227.6:4259 , piece_id=AABEDNUR6UOLDDXWA2QXYRJNH4JD6SPTQLF5ECTYOOUEA6I4PJQA, seg_pos={0 0}, upload=437322]: Done uploading piece. err: <nil> Dec 26 09:45:29.646139234 (498h16m29.51722227s) [noise=true , address=45.91.116.11:4120 , piece_id=ZJ26P7JTMBHT2ZDXUOYJJHRONBY4MUDWJ27LQCXYMDJNEC3LRCPA, seg_pos={0 0}, upload=437322]: Done uploading piece. err: <nil> Dec 26 09:45:29.846764169 (498h16m29.717847204s) [noise=true , address=45.91.116.13:4155 , piece_id=LHTP2PP6OHASBVBOSRZ3HFIMLCESWRER2GYYTV5UEASLDWMEFL2Q, seg_pos={0 0}, upload=437322]: Done uploading piece. err: <nil> Dec 26 09:45:30.422822634 (498h16m30.293905679s) [noise=true , address=45.91.116.5:4299 , piece_id=FAGTZPDEKWLYH35GBJQSMDSCP5TMGQPSPEPGOBNCE73GXDPPF5BA, seg_pos={0 0}, upload=437322]: Done uploading piece. err: <nil> Dec 26 09:45:30.478706595 (498h16m30.349789631s) [noise=true , address=77.90.31.18:4808 , piece_id=J23C4DSZQJPWZMP5ZTH3MYA3H3CDUI5QJX7EPDUPWXRNGWYIK2SA, seg_pos={0 0}, upload=437322]: Done uploading piece. err: <nil> Dec 26 09:45:30.478731243 (498h16m30.349814288s) [seg_pos={0 0}, upload=437322]: Segment reached optimal threshold of 13 pieces. Dec 26 09:45:30.478961211 (498h16m30.350044246s) [noise=true , address=77.90.31.12:4590 , piece_id=EZRVDMSFTVYFFFAB3RW6T45HSMJJCNT3WQAPPWZJMXNO2UNIAHVQ, seg_pos={0 0}, upload=437322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 09:45:30.478968565 (498h16m30.3500516s) [noise=true , address=77.90.31.12:4590 , piece_id=EZRVDMSFTVYFFFAB3RW6T45HSMJJCNT3WQAPPWZJMXNO2UNIAHVQ, seg_pos={0 0}, upload=437322]: Long tail cancellation due to optimal threshold Dec 26 09:45:30.479115995 (498h16m30.35019904s) [noise=true , address=45.91.116.10:4084 , piece_id=WXKRFZKQ7HXCCL3CKU6V2C3O4BUXGE6TZ3B3BQNTQRPBPFXFZ2TA, seg_pos={0 0}, upload=437322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:56574->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:56574->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56574->45.91.116.10:4084: use of closed network connection Dec 26 09:45:30.479125724 (498h16m30.350208759s) [noise=true , address=45.91.116.10:4084 , piece_id=WXKRFZKQ7HXCCL3CKU6V2C3O4BUXGE6TZ3B3BQNTQRPBPFXFZ2TA, seg_pos={0 0}, upload=437322]: Long tail cancellation due to optimal threshold Dec 26 09:45:30.479163275 (498h16m30.35024631s) [seg_pos={0 0}, upload=437322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:45:30.479243769 (498h16m30.350326804s) [upload=437322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:45:30.479259869 (498h16m30.350342904s) [upload=437322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:45:30.765862269 (498h16m30.636945314s) [upload=437322]: Done uploading segments... Dec 26 09:45:30.765888500 (498h16m30.636971535s) [upload=437322]: Upload finished. err: <nil> Duration: 30.003217397s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X......X.] duration: 32.615512679s trace: https://jaeger.dev.storj.io/trace/2340efa351101f24 FAIL (2025-12-26 09:45:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 26 09:44:00.880088265 (498h15m0.751171301s) [noise=true , address=45.91.116.7:4016 , piece_id=JAZZQAOSDSXEAJS57ZNGRUE4RLXQQ6MPBDDJGPD4QNSLU56TZYSQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.880229845 (498h15m0.75131288s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.880757570 (498h15m0.751840615s) [noise=true , address=145.79.227.30:4765 , piece_id=IJRWZ3YBUSGZJ3WJKXEEOWWIFMO25CMBYR2PO537EKHTQHB3XP2Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.881224599 (498h15m0.752307644s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.882867498 (498h15m0.753950533s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.883629057 (498h15m0.754712102s) [noise=true , address=45.91.116.7:4042 , piece_id=QOYKGPE2E7SSX7YCPVHMZYVXOH43MW5ODEJMYEACSUDLPJLP6QHA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.884590658 (498h15m0.755673734s) [noise=true , address=45.91.116.9:4272 , piece_id=4LG7DWD5GJEXX4KGRS6QROVWBXGCUZ4LWOBMIPI2AEK2Q3NIK7XQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.884836046 (498h15m0.755919081s) [noise=true , address=45.91.116.10:4104 , piece_id=5YHM2XVPLW5BUDOFKSRPAOU6FAQEZUFVJQU2PKAXMLSGFBC3L73Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.886094181 (498h15m0.757177236s) [noise=true , address=145.79.227.7:4322 , piece_id=4IXKAMY6AB6HRGNMZLLMZ2WWUUHXRGZGD35VZMORUVG4UHV2Q4PA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.886114190 (498h15m0.757197235s) [noise=true , address=45.91.116.4:4944 , piece_id=PLIUZEYKODIE3JKUFVI43FZ47DYVAE6CGFXT45N6MBOLYJV626UQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.887478658 (498h15m0.758561693s) [noise=true , address=77.90.31.12:4598 , piece_id=L33QFG7QEALMVTS2IOF7DZOA4LCFMKKRPG4E77EALQK2NG622Q2Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.887528633 (498h15m0.758611668s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.888256428 (498h15m0.759339474s) [noise=true , address=45.91.116.14:4195 , piece_id=QEOVODBM24X6ROM2ME3GCTPVUHFHDE5JBWAQSECCRITOAJIFR4SA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.888915033 (498h15m0.759998078s) [noise=true , address=145.79.227.44:4203 , piece_id=6XBUBLM3UGODNA2INBVZLSKVQSQIVKECAYEFMHULVHI4FN7LHMKQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.890072357 (498h15m0.761155392s) [noise=true , address=45.91.116.11:4129 , piece_id=QCGCBLS7PHXGRAB2ZGPUDKL5ADSDMCZHHSJPQVOSPHXMX7O6UIJA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.890947614 (498h15m0.762030659s) [noise=true , address=77.90.31.14:4675 , piece_id=BHLZL54ETYKAACJYLGD3JGOJEIM7IVOS4BIIKDN3ZHDEDBVQBMQA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:02.349863155 (498h15m2.22094619s) [upload=437299]: Next returned nil segment Dec 26 09:44:02.349900766 (498h15m2.220983811s) [upload=437299]: Waiting for error group managing segments... Dec 26 09:44:02.781836534 (498h15m2.65291958s) [noise=true , address=145.79.227.7:4322 , piece_id=4IXKAMY6AB6HRGNMZLLMZ2WWUUHXRGZGD35VZMORUVG4UHV2Q4PA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:02.928532198 (498h15m2.799615243s) [noise=true , address=145.79.227.34:4937 , piece_id=RRTHSCYGQXVBGWD4IATI7XEBM32RDPVQR7T4J3XJQXUIGWU45GCA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:02.959038798 (498h15m2.830121853s) [noise=true , address=77.90.31.14:4675 , piece_id=BHLZL54ETYKAACJYLGD3JGOJEIM7IVOS4BIIKDN3ZHDEDBVQBMQA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.052394553 (498h15m2.923477598s) [noise=true , address=77.90.31.12:4598 , piece_id=L33QFG7QEALMVTS2IOF7DZOA4LCFMKKRPG4E77EALQK2NG622Q2Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.066899663 (498h15m2.937982708s) [noise=true , address=45.91.116.4:4944 , piece_id=PLIUZEYKODIE3JKUFVI43FZ47DYVAE6CGFXT45N6MBOLYJV626UQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.081280326 (498h15m2.952363381s) [noise=true , address=45.91.116.10:4104 , piece_id=5YHM2XVPLW5BUDOFKSRPAOU6FAQEZUFVJQU2PKAXMLSGFBC3L73Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.096434834 (498h15m2.967517869s) [noise=true , address=45.91.116.7:4016 , piece_id=JAZZQAOSDSXEAJS57ZNGRUE4RLXQQ6MPBDDJGPD4QNSLU56TZYSQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.107137242 (498h15m2.978220287s) [noise=true , address=45.91.116.14:4195 , piece_id=QEOVODBM24X6ROM2ME3GCTPVUHFHDE5JBWAQSECCRITOAJIFR4SA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.176790474 (498h15m3.047873519s) [noise=true , address=145.79.227.40:4074 , piece_id=SPEIB4FDHK7I45NIXB3CLX5HNVNKSQZ5APXE4ULGW3MVPEEEKCOA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.431315263 (498h15m3.302398308s) [noise=true , address=45.91.116.9:4272 , piece_id=4LG7DWD5GJEXX4KGRS6QROVWBXGCUZ4LWOBMIPI2AEK2Q3NIK7XQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.710113361 (498h15m3.581196406s) [noise=true , address=145.79.227.44:4203 , piece_id=6XBUBLM3UGODNA2INBVZLSKVQSQIVKECAYEFMHULVHI4FN7LHMKQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.716995455 (498h15m3.5880785s) [noise=true , address=145.79.227.30:4765 , piece_id=IJRWZ3YBUSGZJ3WJKXEEOWWIFMO25CMBYR2PO537EKHTQHB3XP2Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.183858151 (498h15m4.054941206s) [noise=true , address=45.91.116.11:4129 , piece_id=QCGCBLS7PHXGRAB2ZGPUDKL5ADSDMCZHHSJPQVOSPHXMX7O6UIJA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.462428025 (498h15m4.33351106s) [noise=true , address=45.91.116.7:4042 , piece_id=QOYKGPE2E7SSX7YCPVHMZYVXOH43MW5ODEJMYEACSUDLPJLP6QHA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.462455627 (498h15m4.333538662s) [seg_pos={0 0}, upload=437299]: Segment reached optimal threshold of 14 pieces. Dec 26 09:44:04.462751150 (498h15m4.333834195s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection Dec 26 09:44:04.462761229 (498h15m4.333844284s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463065669 (498h15m4.334148724s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463124842 (498h15m4.334207887s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463174215 (498h15m4.334257361s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FNXVNAv3KeiRqTnsvQkMobyRcNFmPAAjVqSNvgs6MPJH4smVU): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463187411 (498h15m4.334270456s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463448007 (498h15m4.334531052s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463458536 (498h15m4.334541582s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463503002 (498h15m4.334586047s) [seg_pos={0 0}, upload=437299]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 09:44:04.463624553 (498h15m4.334707598s) [upload=437299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:44:04.463631466 (498h15m4.334714521s) [upload=437299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:44:04.753033747 (498h15m4.624116782s) [upload=437299]: Done uploading segments... Dec 26 09:44:04.753119550 (498h15m4.624202595s) [upload=437299]: Upload finished. err: <nil> Duration: 32.615530032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....XXX..X.......X.....XX.XX.XXX.....X..X.....X.X...X..XXXXX] duration: 30.002978808s trace: https://jaeger.dev.storj.io/trace/7800d38ac3f43705 FAIL (2025-12-26 09:45:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 53N4LBWUQJC6FSBVD6WI7A5QO2FXIQ7BHKKDT73VFVIGA5XKNZDA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: FU6QXUDPZDLK5WA7TAGNSYFKRGIB2JULP3GNCOB4EI7OQRQYR3NA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: RS4RCDI4MPIO6L65OKTTDK46HGJYP2FZSSSAWIHQFR37W5BCXFIA) context deadline exceeded ======LOGS====== Dec 26 09:44:56.789546740 (498h15m56.660629775s) [upload=437313]: Starting upload Dec 26 09:44:56.789596745 (498h15m56.66067978s) [upload=437313]: Done starting upload Dec 26 09:44:56.789656639 (498h15m56.660739674s) [upload=437313]: Uploading segments... Dec 26 09:44:56.789895644 (498h15m56.660978679s) [upload=437313]: Got next segment. Inline: false Dec 26 09:44:56.789936612 (498h15m56.661019647s) [upload=437313]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:44:56.789942062 (498h15m56.661025097s) [upload=437313]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:44:57.034751843 (498h15m56.905834878s) [seg_pos={0 0}, upload=437313]: Begin upload segment... Dec 26 09:44:57.034801527 (498h15m56.905884562s) [seg_pos={0 0}, upload=437313]: Done begin upload segment. Dec 26 09:44:57.034829731 (498h15m56.905912766s) [noise=true , address=145.79.227.11:4729 , piece_id=FHUOTWGPZK4FNB7SYMN5NMDJLRQDLWCKJ7CENYBU5Y3HFXZFU5UQ, seg_pos={0 0}, upload=437313]: Uploading piece... Dec 26 09:44:57.034844670 (498h15m56.905927705s) [noise=true , address=145.79.227.13:4804 , piece_id=HXLUIXGGOW745DQDPIILKEYONNKPGA36ZNRWIBQZDQD6YJEP7SAQ, seg_pos={0 0}, upload=437313]: Uploading piece... Dec 26 09:44:57.035328320 (498h15m56.906411365s) [noise=true , address=45.91.116.13:4156 , piece_id=53N4LBWUQJC6FSBVD6WI7A5QO2FXIQ7BHKKDT73VFVIGA5XKNZDA, seg_pos={0 0}, upload=437313]: Uploading piece... Dec 26 09:44:57.035730266 (498h15m56.906813311s) [noise=true , address=145.79.227.43:4157 , piece_id=4TTI4S2IGW32B3FSINRBPYDRKQJOGGJL6YH4NPEQ5PUZQXPRYKEA, seg_pos={0 0}, upload=437313]: Uploading piece... Dec 26 09:44:57.036135808 (498h15m56.907218843s) [noise=true , address=145.79.227.42:4122 , piece_id=BMMPP6J5DB3JCOMFTCZ3KVSE6AV6OPYCFO7URPOUR2SZESB6XVXQ, seg_pos={0 0}, upload=437313]: Uploading piece... Dec 26 09:44:57.036501474 (498h15m56.907584529s) [noise=true , address=45.91.116.8:4047 , piece_id=FU6QXUDPZDLK5WA7TAGNSYFKRGIB2JULP3GNCOB4EI7OQRQYR3NA, seg_pos={0 0}, upload=437313]: Uploading piece... Dec 26 09:44:57.036912206 (498h15m56.907995241s) [noise=true , address=77.90.31.10:4517 , piece_id=NHKCI6TOO4LEJV255XGBLKT7Z2MPGP3P4CJSEZB6RW7VGLH3M3MQ, seg_pos={0 0}, upload=437313]: Uploading piece... Dec 26 09:44:57.038886095 (498h15m56.90996913s) [noise=true , address=45.91.116.9:4264 , piece_id=QXLMQPMNCCQRGQUYKUZ3JUIF2KT6BUXC7WQACSPZLRW2JNQEXZAQ, seg_pos={0 0}, upload=437313]: Uploading piece... Dec 26 09:44:57.040053347 (498h15m56.911136392s) [noise=true , address=77.90.31.13:4629 , piece_id=RS4RCDI4MPIO6L65OKTTDK46HGJYP2FZSSSAWIHQFR37W5BCXFIA, seg_pos={0 0}, upload=437313]: Uploading piece... Dec 26 09:44:57.041328034 (498h15m56.912411079s) [noise=true , address=145.79.227.22:4511 , piece_id=C5WVLS2W3I4Z64XTDHTO7OA3P2QUL5MD3KLOZ3LIJ4Y7UELRKTGA, seg_pos={0 0}, upload=437313]: Uploading piece... Dec 26 09:44:57.042829653 (498h15m56.913912688s) [noise=true , address=145.79.227.29:4941 , piece_id=HDQZMPI35IBDUKUT6H7VN4NWDLOXWVEAB7DBY3RC5LWAEUOR6XKQ, seg_pos={0 0}, upload=437313]: Uploading piece... Dec 26 09:44:57.043983511 (498h15m56.915066546s) [noise=true , address=145.79.227.48:4301 , piece_id=XS3LD7YMYTUUQOGKCNUO2J7BLGGRNAR5FPL2C2WRG3WNYFXJQWPQ, seg_pos={0 0}, upload=437313]: Uploading piece... Dec 26 09:44:57.045302281 (498h15m56.916385327s) [noise=true , address=145.79.227.47:4266 , piece_id=QKZMXGB6YTNRDV6M6HY7AE3RVLEKOOU4QE6IAFICPTGEV5YWPZ6Q, seg_pos={0 0}, upload=437313]: Uploading piece... Dec 26 09:44:57.045472516 (498h15m56.916555551s) [noise=true , address=145.79.227.24:4583 , piece_id=HFKJFGCWZGMQG6I5QNIMPNH6JHXZXDCOP3M6INSDO6GVZT2IS2RA, seg_pos={0 0}, upload=437313]: Uploading piece... Dec 26 09:44:57.046683512 (498h15m56.917766547s) [noise=true , address=145.79.227.44:4194 , piece_id=3S2AU4B5PASV4NNJPN4JEBODZI3BLQCRUX3GIRNRXV77AUKKEO6A, seg_pos={0 0}, upload=437313]: Uploading piece... Dec 26 09:44:57.046870929 (498h15m56.917953974s) [noise=true , address=77.90.31.15:4697 , piece_id=T73XTXCTSZSC67G4V6CJGNABSBPITNKPY7G55JQKAZPKRB5AYUMA, seg_pos={0 0}, upload=437313]: Uploading piece... Dec 26 09:44:58.145954069 (498h15m58.017037114s) [noise=true , address=77.90.31.15:4697 , piece_id=T73XTXCTSZSC67G4V6CJGNABSBPITNKPY7G55JQKAZPKRB5AYUMA, seg_pos={0 0}, upload=437313]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 26 09:44:58.751356658 (498h15m58.622439703s) [upload=437313]: Next returned nil segment Dec 26 09:44:58.751370825 (498h15m58.62245386s) [upload=437313]: Waiting for error group managing segments... Dec 26 09:44:59.039104609 (498h15m58.910187654s) [noise=true , address=145.79.227.11:4729 , piece_id=FHUOTWGPZK4FNB7SYMN5NMDJLRQDLWCKJ7CENYBU5Y3HFXZFU5UQ, seg_pos={0 0}, upload=437313]: Done uploading piece. err: <nil> Dec 26 09:44:59.039437994 (498h15m58.910521039s) [noise=true , address=145.79.227.24:4583 , piece_id=HFKJFGCWZGMQG6I5QNIMPNH6JHXZXDCOP3M6INSDO6GVZT2IS2RA, seg_pos={0 0}, upload=437313]: Done uploading piece. err: <nil> Dec 26 09:44:59.047682152 (498h15m58.918765197s) [noise=true , address=145.79.227.42:4122 , piece_id=BMMPP6J5DB3JCOMFTCZ3KVSE6AV6OPYCFO7URPOUR2SZESB6XVXQ, seg_pos={0 0}, upload=437313]: Done uploading piece. err: <nil> Dec 26 09:44:59.048342689 (498h15m58.919425734s) [noise=true , address=145.79.227.43:4157 , piece_id=4TTI4S2IGW32B3FSINRBPYDRKQJOGGJL6YH4NPEQ5PUZQXPRYKEA, seg_pos={0 0}, upload=437313]: Done uploading piece. err: <nil> Dec 26 09:44:59.063520691 (498h15m58.934603736s) [noise=true , address=145.79.227.13:4804 , piece_id=HXLUIXGGOW745DQDPIILKEYONNKPGA36ZNRWIBQZDQD6YJEP7SAQ, seg_pos={0 0}, upload=437313]: Done uploading piece. err: <nil> Dec 26 09:44:59.066960852 (498h15m58.938043897s) [noise=true , address=145.79.227.48:4301 , piece_id=XS3LD7YMYTUUQOGKCNUO2J7BLGGRNAR5FPL2C2WRG3WNYFXJQWPQ, seg_pos={0 0}, upload=437313]: Done uploading piece. err: <nil> Dec 26 09:44:59.069417881 (498h15m58.940500926s) [noise=true , address=145.79.227.29:4941 , piece_id=HDQZMPI35IBDUKUT6H7VN4NWDLOXWVEAB7DBY3RC5LWAEUOR6XKQ, seg_pos={0 0}, upload=437313]: Done uploading piece. err: <nil> Dec 26 09:44:59.070451558 (498h15m58.941534603s) [noise=true , address=145.79.227.47:4266 , piece_id=QKZMXGB6YTNRDV6M6HY7AE3RVLEKOOU4QE6IAFICPTGEV5YWPZ6Q, seg_pos={0 0}, upload=437313]: Done uploading piece. err: <nil> Dec 26 09:44:59.072875304 (498h15m58.943958339s) [noise=true , address=145.79.227.22:4511 , piece_id=C5WVLS2W3I4Z64XTDHTO7OA3P2QUL5MD3KLOZ3LIJ4Y7UELRKTGA, seg_pos={0 0}, upload=437313]: Done uploading piece. err: <nil> Dec 26 09:44:59.281951019 (498h15m59.153034064s) [noise=true , address=45.91.116.8:4047 , piece_id=FU6QXUDPZDLK5WA7TAGNSYFKRGIB2JULP3GNCOB4EI7OQRQYR3NA, seg_pos={0 0}, upload=437313]: Done uploading piece. err: <nil> Dec 26 09:44:59.305004988 (498h15m59.176088023s) [noise=true , address=45.91.116.13:4156 , piece_id=53N4LBWUQJC6FSBVD6WI7A5QO2FXIQ7BHKKDT73VFVIGA5XKNZDA, seg_pos={0 0}, upload=437313]: Done uploading piece. err: <nil> Dec 26 09:44:59.340147595 (498h15m59.21123064s) [noise=true , address=77.90.31.13:4629 , piece_id=RS4RCDI4MPIO6L65OKTTDK46HGJYP2FZSSSAWIHQFR37W5BCXFIA, seg_pos={0 0}, upload=437313]: Done uploading piece. err: <nil> Dec 26 09:45:00.036715538 (498h15m59.907798603s) [noise=true , address=45.91.116.9:4264 , piece_id=QXLMQPMNCCQRGQUYKUZ3JUIF2KT6BUXC7WQACSPZLRW2JNQEXZAQ, seg_pos={0 0}, upload=437313]: Done uploading piece. err: <nil> Dec 26 09:45:00.036741137 (498h15m59.907824182s) [seg_pos={0 0}, upload=437313]: Segment reached optimal threshold of 13 pieces. Dec 26 09:45:00.036972938 (498h15m59.908056003s) [noise=true , address=77.90.31.10:4517 , piece_id=NHKCI6TOO4LEJV255XGBLKT7Z2MPGP3P4CJSEZB6RW7VGLH3M3MQ, seg_pos={0 0}, upload=437313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:53036->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:53036->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53036->77.90.31.10:4517: use of closed network connection Dec 26 09:45:00.036984200 (498h15m59.908067235s) [noise=true , address=77.90.31.10:4517 , piece_id=NHKCI6TOO4LEJV255XGBLKT7Z2MPGP3P4CJSEZB6RW7VGLH3M3MQ, seg_pos={0 0}, upload=437313]: Long tail cancellation due to optimal threshold Dec 26 09:45:00.037037992 (498h15m59.908121057s) [noise=true , address=145.79.227.44:4194 , piece_id=3S2AU4B5PASV4NNJPN4JEBODZI3BLQCRUX3GIRNRXV77AUKKEO6A, seg_pos={0 0}, upload=437313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 09:45:00.037047870 (498h15m59.908130905s) [noise=true , address=145.79.227.44:4194 , piece_id=3S2AU4B5PASV4NNJPN4JEBODZI3BLQCRUX3GIRNRXV77AUKKEO6A, seg_pos={0 0}, upload=437313]: Long tail cancellation due to optimal threshold Dec 26 09:45:00.037079390 (498h15m59.908162466s) [seg_pos={0 0}, upload=437313]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:45:00.037247060 (498h15m59.908330115s) [upload=437313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:45:00.037255105 (498h15m59.90833815s) [upload=437313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:45:00.332553373 (498h16m0.203636408s) [upload=437313]: Done uploading segments... Dec 26 09:45:00.332581467 (498h16m0.203664502s) [upload=437313]: Upload finished. err: <nil> Duration: 30.003003465s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X......X] duration: 1m0.002011378s trace: https://jaeger.dev.storj.io/trace/2340efa351101f24 FAIL (2025-12-26 09:44:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 26 09:44:00.880088265 (498h15m0.751171301s) [noise=true , address=45.91.116.7:4016 , piece_id=JAZZQAOSDSXEAJS57ZNGRUE4RLXQQ6MPBDDJGPD4QNSLU56TZYSQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.880229845 (498h15m0.75131288s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.880757570 (498h15m0.751840615s) [noise=true , address=145.79.227.30:4765 , piece_id=IJRWZ3YBUSGZJ3WJKXEEOWWIFMO25CMBYR2PO537EKHTQHB3XP2Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.881224599 (498h15m0.752307644s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.882867498 (498h15m0.753950533s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.883629057 (498h15m0.754712102s) [noise=true , address=45.91.116.7:4042 , piece_id=QOYKGPE2E7SSX7YCPVHMZYVXOH43MW5ODEJMYEACSUDLPJLP6QHA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.884590658 (498h15m0.755673734s) [noise=true , address=45.91.116.9:4272 , piece_id=4LG7DWD5GJEXX4KGRS6QROVWBXGCUZ4LWOBMIPI2AEK2Q3NIK7XQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.884836046 (498h15m0.755919081s) [noise=true , address=45.91.116.10:4104 , piece_id=5YHM2XVPLW5BUDOFKSRPAOU6FAQEZUFVJQU2PKAXMLSGFBC3L73Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.886094181 (498h15m0.757177236s) [noise=true , address=145.79.227.7:4322 , piece_id=4IXKAMY6AB6HRGNMZLLMZ2WWUUHXRGZGD35VZMORUVG4UHV2Q4PA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.886114190 (498h15m0.757197235s) [noise=true , address=45.91.116.4:4944 , piece_id=PLIUZEYKODIE3JKUFVI43FZ47DYVAE6CGFXT45N6MBOLYJV626UQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.887478658 (498h15m0.758561693s) [noise=true , address=77.90.31.12:4598 , piece_id=L33QFG7QEALMVTS2IOF7DZOA4LCFMKKRPG4E77EALQK2NG622Q2Q, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.887528633 (498h15m0.758611668s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.888256428 (498h15m0.759339474s) [noise=true , address=45.91.116.14:4195 , piece_id=QEOVODBM24X6ROM2ME3GCTPVUHFHDE5JBWAQSECCRITOAJIFR4SA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.888915033 (498h15m0.759998078s) [noise=true , address=145.79.227.44:4203 , piece_id=6XBUBLM3UGODNA2INBVZLSKVQSQIVKECAYEFMHULVHI4FN7LHMKQ, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.890072357 (498h15m0.761155392s) [noise=true , address=45.91.116.11:4129 , piece_id=QCGCBLS7PHXGRAB2ZGPUDKL5ADSDMCZHHSJPQVOSPHXMX7O6UIJA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:00.890947614 (498h15m0.762030659s) [noise=true , address=77.90.31.14:4675 , piece_id=BHLZL54ETYKAACJYLGD3JGOJEIM7IVOS4BIIKDN3ZHDEDBVQBMQA, seg_pos={0 0}, upload=437299]: Uploading piece... Dec 26 09:44:02.349863155 (498h15m2.22094619s) [upload=437299]: Next returned nil segment Dec 26 09:44:02.349900766 (498h15m2.220983811s) [upload=437299]: Waiting for error group managing segments... Dec 26 09:44:02.781836534 (498h15m2.65291958s) [noise=true , address=145.79.227.7:4322 , piece_id=4IXKAMY6AB6HRGNMZLLMZ2WWUUHXRGZGD35VZMORUVG4UHV2Q4PA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:02.928532198 (498h15m2.799615243s) [noise=true , address=145.79.227.34:4937 , piece_id=RRTHSCYGQXVBGWD4IATI7XEBM32RDPVQR7T4J3XJQXUIGWU45GCA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:02.959038798 (498h15m2.830121853s) [noise=true , address=77.90.31.14:4675 , piece_id=BHLZL54ETYKAACJYLGD3JGOJEIM7IVOS4BIIKDN3ZHDEDBVQBMQA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.052394553 (498h15m2.923477598s) [noise=true , address=77.90.31.12:4598 , piece_id=L33QFG7QEALMVTS2IOF7DZOA4LCFMKKRPG4E77EALQK2NG622Q2Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.066899663 (498h15m2.937982708s) [noise=true , address=45.91.116.4:4944 , piece_id=PLIUZEYKODIE3JKUFVI43FZ47DYVAE6CGFXT45N6MBOLYJV626UQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.081280326 (498h15m2.952363381s) [noise=true , address=45.91.116.10:4104 , piece_id=5YHM2XVPLW5BUDOFKSRPAOU6FAQEZUFVJQU2PKAXMLSGFBC3L73Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.096434834 (498h15m2.967517869s) [noise=true , address=45.91.116.7:4016 , piece_id=JAZZQAOSDSXEAJS57ZNGRUE4RLXQQ6MPBDDJGPD4QNSLU56TZYSQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.107137242 (498h15m2.978220287s) [noise=true , address=45.91.116.14:4195 , piece_id=QEOVODBM24X6ROM2ME3GCTPVUHFHDE5JBWAQSECCRITOAJIFR4SA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.176790474 (498h15m3.047873519s) [noise=true , address=145.79.227.40:4074 , piece_id=SPEIB4FDHK7I45NIXB3CLX5HNVNKSQZ5APXE4ULGW3MVPEEEKCOA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.431315263 (498h15m3.302398308s) [noise=true , address=45.91.116.9:4272 , piece_id=4LG7DWD5GJEXX4KGRS6QROVWBXGCUZ4LWOBMIPI2AEK2Q3NIK7XQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.710113361 (498h15m3.581196406s) [noise=true , address=145.79.227.44:4203 , piece_id=6XBUBLM3UGODNA2INBVZLSKVQSQIVKECAYEFMHULVHI4FN7LHMKQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:03.716995455 (498h15m3.5880785s) [noise=true , address=145.79.227.30:4765 , piece_id=IJRWZ3YBUSGZJ3WJKXEEOWWIFMO25CMBYR2PO537EKHTQHB3XP2Q, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.183858151 (498h15m4.054941206s) [noise=true , address=45.91.116.11:4129 , piece_id=QCGCBLS7PHXGRAB2ZGPUDKL5ADSDMCZHHSJPQVOSPHXMX7O6UIJA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.462428025 (498h15m4.33351106s) [noise=true , address=45.91.116.7:4042 , piece_id=QOYKGPE2E7SSX7YCPVHMZYVXOH43MW5ODEJMYEACSUDLPJLP6QHA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: <nil> Dec 26 09:44:04.462455627 (498h15m4.333538662s) [seg_pos={0 0}, upload=437299]: Segment reached optimal threshold of 14 pieces. Dec 26 09:44:04.462751150 (498h15m4.333834195s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47892->64.49.13.169:4706: use of closed network connection Dec 26 09:44:04.462761229 (498h15m4.333844284s) [noise=true , address=64.49.13.169:4706 , piece_id=WR35DNJHVDBEO263JR2JNRWK4VUOPDNKMFTVZLUXHLCAOLGUHG6A, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463065669 (498h15m4.334148724s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463124842 (498h15m4.334207887s) [noise=true , address=77.90.31.19:4856 , piece_id=CBOXORD34I35VGN22HORXJU5E3H5LRKY5SCN5IRRNLNH6HWT2SNA, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463174215 (498h15m4.334257361s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FNXVNAv3KeiRqTnsvQkMobyRcNFmPAAjVqSNvgs6MPJH4smVU): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463187411 (498h15m4.334270456s) [noise=true , address=77.90.31.13:4650 , piece_id=3G322BNEGEHM32R4XCSWZVX4MV5W3QQBYYOJVQUPHT3C3QWBS2IA, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463448007 (498h15m4.334531052s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:04.463458536 (498h15m4.334541582s) [noise=true , address=77.90.31.21:4886 , piece_id=R4NGF7H6S7ZX3A2V3HGS7GNCZEW7LHNMUCTJAXBNCH3WKSYGM6OQ, seg_pos={0 0}, upload=437299]: Long tail cancellation due to optimal threshold Dec 26 09:44:04.463503002 (498h15m4.334586047s) [seg_pos={0 0}, upload=437299]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 09:44:04.463624553 (498h15m4.334707598s) [upload=437299]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:44:04.463631466 (498h15m4.334714521s) [upload=437299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:44:04.753033747 (498h15m4.624116782s) [upload=437299]: Done uploading segments... Dec 26 09:44:04.753119550 (498h15m4.624202595s) [upload=437299]: Upload finished. err: <nil> Duration: 1m0.002020635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X....XXX..X.......X.....XX.XX.XXX.....X..X.....X.X...X..XXXX] duration: 30.004038009s trace: https://jaeger.dev.storj.io/trace/4d2e744dfd7d0848 FAIL (2025-12-26 09:44:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: XJXCMCU4CPJTPLSZRAJYE75KJYCMHVLGZJY7WZF2FK7YJXKGF5BQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: M47GSXD6DRXII64G2D4GTIDYRHVRZN2GQNDJXNPXCY6CJPUXZGEQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: JVB6RVCC7U7BZ3M27IRJWZXESPAUQEXLZEMBQHUHTEKTAEJKDCGA) context deadline exceeded ======LOGS====== Dec 26 09:44:02.943739896 (498h15m2.814822942s) [upload=437300]: Starting upload Dec 26 09:44:02.943862650 (498h15m2.814945705s) [upload=437300]: Done starting upload Dec 26 09:44:02.944027434 (498h15m2.815110479s) [upload=437300]: Uploading segments... Dec 26 09:44:02.944196175 (498h15m2.81527922s) [upload=437300]: Got next segment. Inline: false Dec 26 09:44:02.944237714 (498h15m2.815320779s) [upload=437300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:44:02.944245849 (498h15m2.815328894s) [upload=437300]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:44:03.187169808 (498h15m3.058252853s) [seg_pos={0 0}, upload=437300]: Begin upload segment... Dec 26 09:44:03.187218582 (498h15m3.058301627s) [seg_pos={0 0}, upload=437300]: Done begin upload segment. Dec 26 09:44:03.187256023 (498h15m3.058339068s) [noise=true , address=145.79.227.22:4511 , piece_id=RO3YMHZ4ZVGXXCGPYDK7Q34BNTUOIRVFMI5ZKPB644IBQ57NXCDQ, seg_pos={0 0}, upload=437300]: Uploading piece... Dec 26 09:44:03.187660182 (498h15m3.058743227s) [noise=true , address=145.79.227.39:4013 , piece_id=DQ5SP2WKE3JPHSK4Z44MLLGZR3IQ4ZBZCRNOAGOH7WGQ2XJYTPFQ, seg_pos={0 0}, upload=437300]: Uploading piece... Dec 26 09:44:03.188038472 (498h15m3.059121507s) [noise=true , address=45.91.116.10:4084 , piece_id=ZIRZF6VVRELEGK7WTLU3Q4SLUFUZG5OFJA5QSQ44DDI6ZDXUVF6A, seg_pos={0 0}, upload=437300]: Uploading piece... Dec 26 09:44:03.188869144 (498h15m3.059952189s) [noise=true , address=45.91.116.11:4119 , piece_id=JVB6RVCC7U7BZ3M27IRJWZXESPAUQEXLZEMBQHUHTEKTAEJKDCGA, seg_pos={0 0}, upload=437300]: Uploading piece... Dec 26 09:44:03.189250991 (498h15m3.060334026s) [noise=true , address=77.90.31.19:4842 , piece_id=NRF6TH4HVSRELLSKEWT2DBUARHAGQPDKLKQIOSEF7NXRFEDAA22Q, seg_pos={0 0}, upload=437300]: Uploading piece... Dec 26 09:44:03.189671732 (498h15m3.060754777s) [noise=true , address=45.91.116.7:4011 , piece_id=XJXCMCU4CPJTPLSZRAJYE75KJYCMHVLGZJY7WZF2FK7YJXKGF5BQ, seg_pos={0 0}, upload=437300]: Uploading piece... Dec 26 09:44:03.189748518 (498h15m3.060831553s) [noise=true , address=145.79.227.29:4943 , piece_id=3Y6MAHNNB7PT2MNFB2XS6W4P5JSTC6H5R3CQZQYPVLUX5CIRSHJQ, seg_pos={0 0}, upload=437300]: Uploading piece... Dec 26 09:44:03.190101029 (498h15m3.061184074s) [noise=true , address=77.90.31.12:4590 , piece_id=M47GSXD6DRXII64G2D4GTIDYRHVRZN2GQNDJXNPXCY6CJPUXZGEQ, seg_pos={0 0}, upload=437300]: Uploading piece... Dec 26 09:44:03.190418985 (498h15m3.06150203s) [noise=true , address=45.91.116.14:4192 , piece_id=UKK5PMXMHMZU3G26V3KIS6G27MBFJHQGJ5L3FQJIZZ2BF3KBPXCQ, seg_pos={0 0}, upload=437300]: Uploading piece... Dec 26 09:44:03.190586384 (498h15m3.061669429s) [noise=true , address=145.79.227.11:4731 , piece_id=VAYJ5UM7BHQBUBJTDATB6OFH2QJNCCK3VAYTE3LJN5HTHNJV45PA, seg_pos={0 0}, upload=437300]: Uploading piece... Dec 26 09:44:03.190834386 (498h15m3.061917431s) [noise=true , address=145.79.227.17:4970 , piece_id=BDUPCQD6K4XXO5TX7KXKX4KF4UJULOMQK3KXQQ2NTZRIYJTDFM7A, seg_pos={0 0}, upload=437300]: Uploading piece... Dec 26 09:44:03.190910692 (498h15m3.061993727s) [noise=true , address=45.91.116.9:4261 , piece_id=OZKXAK4NY4HIWSGIV2LDINLTX2OXOKJ6QUMDFBH4YIYTKBZU44WA, seg_pos={0 0}, upload=437300]: Uploading piece... Dec 26 09:44:03.191351992 (498h15m3.062435037s) [noise=true , address=145.79.227.36:4517 , piece_id=6IKWYRZJS26BBLYPWISM2K3V76SA4LSNM2K5EE6FQCST55YLABEA, seg_pos={0 0}, upload=437300]: Uploading piece... Dec 26 09:44:03.191419650 (498h15m3.062502695s) [noise=true , address=45.91.116.6:4976 , piece_id=4MJ6CQ4CL22QLS2J5GFTJHBZ2FJVSWIHY7DHMNNXH2PQG52H6POA, seg_pos={0 0}, upload=437300]: Uploading piece... Dec 26 09:44:03.191834651 (498h15m3.062917696s) [noise=true , address=145.79.227.6:4258 , piece_id=SR4YDWFVS7KHZYTBRHBUUKR5W43Q7HWPGOT6FBNX26BJ5W6SGRCQ, seg_pos={0 0}, upload=437300]: Uploading piece... Dec 26 09:44:03.192500468 (498h15m3.063583513s) [noise=true , address=145.79.227.13:4804 , piece_id=QVZJJYCIRJXEZWTQPUJHQZJIV27ESGGOBMVSURMBMSTNJROCRTNA, seg_pos={0 0}, upload=437300]: Uploading piece... Dec 26 09:44:04.809309295 (498h15m4.68039234s) [upload=437300]: Next returned nil segment Dec 26 09:44:04.809355974 (498h15m4.68043903s) [upload=437300]: Waiting for error group managing segments... Dec 26 09:44:05.194725563 (498h15m5.065808608s) [noise=true , address=145.79.227.11:4731 , piece_id=VAYJ5UM7BHQBUBJTDATB6OFH2QJNCCK3VAYTE3LJN5HTHNJV45PA, seg_pos={0 0}, upload=437300]: Done uploading piece. err: <nil> Dec 26 09:44:05.198037029 (498h15m5.069120064s) [noise=true , address=145.79.227.17:4970 , piece_id=BDUPCQD6K4XXO5TX7KXKX4KF4UJULOMQK3KXQQ2NTZRIYJTDFM7A, seg_pos={0 0}, upload=437300]: Done uploading piece. err: <nil> Dec 26 09:44:05.200008222 (498h15m5.071091257s) [noise=true , address=145.79.227.13:4804 , piece_id=QVZJJYCIRJXEZWTQPUJHQZJIV27ESGGOBMVSURMBMSTNJROCRTNA, seg_pos={0 0}, upload=437300]: Done uploading piece. err: <nil> Dec 26 09:44:05.200575983 (498h15m5.071659028s) [noise=true , address=145.79.227.39:4013 , piece_id=DQ5SP2WKE3JPHSK4Z44MLLGZR3IQ4ZBZCRNOAGOH7WGQ2XJYTPFQ, seg_pos={0 0}, upload=437300]: Done uploading piece. err: <nil> Dec 26 09:44:05.200861146 (498h15m5.071944191s) [noise=true , address=145.79.227.36:4517 , piece_id=6IKWYRZJS26BBLYPWISM2K3V76SA4LSNM2K5EE6FQCST55YLABEA, seg_pos={0 0}, upload=437300]: Done uploading piece. err: <nil> Dec 26 09:44:05.213243545 (498h15m5.08432659s) [noise=true , address=145.79.227.22:4511 , piece_id=RO3YMHZ4ZVGXXCGPYDK7Q34BNTUOIRVFMI5ZKPB644IBQ57NXCDQ, seg_pos={0 0}, upload=437300]: Done uploading piece. err: <nil> Dec 26 09:44:05.216616347 (498h15m5.087699392s) [noise=true , address=145.79.227.29:4943 , piece_id=3Y6MAHNNB7PT2MNFB2XS6W4P5JSTC6H5R3CQZQYPVLUX5CIRSHJQ, seg_pos={0 0}, upload=437300]: Done uploading piece. err: <nil> Dec 26 09:44:05.219230555 (498h15m5.09031359s) [noise=true , address=145.79.227.6:4258 , piece_id=SR4YDWFVS7KHZYTBRHBUUKR5W43Q7HWPGOT6FBNX26BJ5W6SGRCQ, seg_pos={0 0}, upload=437300]: Done uploading piece. err: <nil> Dec 26 09:44:05.660340402 (498h15m5.531423457s) [noise=true , address=77.90.31.12:4590 , piece_id=M47GSXD6DRXII64G2D4GTIDYRHVRZN2GQNDJXNPXCY6CJPUXZGEQ, seg_pos={0 0}, upload=437300]: Done uploading piece. err: <nil> Dec 26 09:44:05.776139333 (498h15m5.647222368s) [noise=true , address=77.90.31.19:4842 , piece_id=NRF6TH4HVSRELLSKEWT2DBUARHAGQPDKLKQIOSEF7NXRFEDAA22Q, seg_pos={0 0}, upload=437300]: Done uploading piece. err: <nil> Dec 26 09:44:06.230608388 (498h15m6.101691423s) [noise=true , address=45.91.116.11:4119 , piece_id=JVB6RVCC7U7BZ3M27IRJWZXESPAUQEXLZEMBQHUHTEKTAEJKDCGA, seg_pos={0 0}, upload=437300]: Done uploading piece. err: <nil> Dec 26 09:44:06.320614706 (498h15m6.191697741s) [noise=true , address=45.91.116.7:4011 , piece_id=XJXCMCU4CPJTPLSZRAJYE75KJYCMHVLGZJY7WZF2FK7YJXKGF5BQ, seg_pos={0 0}, upload=437300]: Done uploading piece. err: <nil> Dec 26 09:44:06.488271192 (498h15m6.359354237s) [noise=true , address=45.91.116.9:4261 , piece_id=OZKXAK4NY4HIWSGIV2LDINLTX2OXOKJ6QUMDFBH4YIYTKBZU44WA, seg_pos={0 0}, upload=437300]: Done uploading piece. err: <nil> Dec 26 09:44:06.488289708 (498h15m6.359372753s) [seg_pos={0 0}, upload=437300]: Segment reached optimal threshold of 13 pieces. Dec 26 09:44:06.488466735 (498h15m6.35954978s) [noise=true , address=45.91.116.10:4084 , piece_id=ZIRZF6VVRELEGK7WTLU3Q4SLUFUZG5OFJA5QSQ44DDI6ZDXUVF6A, seg_pos={0 0}, upload=437300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:06.488474600 (498h15m6.359557635s) [noise=true , address=45.91.116.10:4084 , piece_id=ZIRZF6VVRELEGK7WTLU3Q4SLUFUZG5OFJA5QSQ44DDI6ZDXUVF6A, seg_pos={0 0}, upload=437300]: Long tail cancellation due to optimal threshold Dec 26 09:44:06.488576373 (498h15m6.359659418s) [noise=true , address=45.91.116.6:4976 , piece_id=4MJ6CQ4CL22QLS2J5GFTJHBZ2FJVSWIHY7DHMNNXH2PQG52H6POA, seg_pos={0 0}, upload=437300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:06.488585270 (498h15m6.359668305s) [noise=true , address=45.91.116.6:4976 , piece_id=4MJ6CQ4CL22QLS2J5GFTJHBZ2FJVSWIHY7DHMNNXH2PQG52H6POA, seg_pos={0 0}, upload=437300]: Long tail cancellation due to optimal threshold Dec 26 09:44:06.488691362 (498h15m6.359774407s) [noise=true , address=45.91.116.14:4192 , piece_id=UKK5PMXMHMZU3G26V3KIS6G27MBFJHQGJ5L3FQJIZZ2BF3KBPXCQ, seg_pos={0 0}, upload=437300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 26 09:44:06.488699037 (498h15m6.359782083s) [noise=true , address=45.91.116.14:4192 , piece_id=UKK5PMXMHMZU3G26V3KIS6G27MBFJHQGJ5L3FQJIZZ2BF3KBPXCQ, seg_pos={0 0}, upload=437300]: Long tail cancellation due to optimal threshold Dec 26 09:44:06.488715037 (498h15m6.359798083s) [seg_pos={0 0}, upload=437300]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:44:06.488836780 (498h15m6.359919825s) [upload=437300]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:44:06.488845155 (498h15m6.359928191s) [upload=437300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:44:06.777167139 (498h15m6.648250174s) [upload=437300]: Done uploading segments... Dec 26 09:44:06.777189622 (498h15m6.648272667s) [upload=437300]: Upload finished. err: <nil> Duration: 30.004058768s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 38.769181935s trace: https://jaeger.dev.storj.io/trace/667977214d97ec8c FAIL (2025-12-26 09:43:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, Piece ID: YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA) context deadline exceeded; piecestore: (Node ID: 1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, Piece ID: MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A) context deadline exceeded ======LOGS====== true , address=77.90.31.11:4566 , piece_id=L4EZMVP56QYGCJPGWK3FQ4NO5HO7Y4KCRFWKEG2TWATBFEFIGUIA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.357693223 (498h9m2.228776258s) [noise=true , address=77.90.31.21:4886 , piece_id=HSVV4VVNNYY6KZR2XPAHYDVPJO6PPBMMRS4JVEOWMZAIVDDNWJGQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.358418583 (498h9m2.229501628s) [noise=true , address=64.49.13.140:4508 , piece_id=64FDQH7EMUMDSWRGXSCBD63EGZACEJKZJLXL4IYKSP7RPZHLALNA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.358935137 (498h9m2.230018182s) [noise=true , address=77.90.31.16:4756 , piece_id=BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359491216 (498h9m2.230574261s) [noise=true , address=45.91.116.9:4288 , piece_id=MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359953967 (498h9m2.231037013s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359968154 (498h9m2.231051199s) [noise=true , address=77.90.31.18:4813 , piece_id=LMQL4MOLI2UM2AGA774XJY3FFU2C4OC66TVEBNZKRWQPGZNJ2GYA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.360429653 (498h9m2.231512698s) [noise=true , address=145.79.227.40:4081 , piece_id=FQKTTQ4RF3CL4Z7ASHOZFINUSEKUSC3WELUAJHUK5NWQPZZ5NUQA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.360518873 (498h9m2.231601918s) [noise=true , address=145.79.227.7:4322 , piece_id=EKN2EMS5DQCO6QZNIPUSTJIOHB6S5ZXD4NCJCUW2UGRYSGZRUEYQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361114697 (498h9m2.232197742s) [noise=true , address=45.91.116.8:4059 , piece_id=VGGWQT6LBVG3I2PGQE2CGNSUETQ6PGPFYWYCGULPF2YYZB4YGUNA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361136940 (498h9m2.232219985s) [noise=true , address=45.91.116.5:4330 , piece_id=YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361631942 (498h9m2.232714988s) [noise=true , address=45.91.116.10:4100 , piece_id=SIU7MB27XG7QQWM2ZDUADCNCBKJO32TCGJPRUETYKUDZORZK32UA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361832193 (498h9m2.232915238s) [noise=true , address=145.79.227.34:4933 , piece_id=7ODKEDJBK3WMG4LHBPVJE2PGMURUMQBHGMA7J2CVNXP6DK5DIL6A, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.362940474 (498h9m2.234023529s) [noise=true , address=145.79.227.44:4225 , piece_id=IUI7KC75SGIV7JY2G6KS6I3PAEJ7GOHL6ZJH5F6XQSLSE3YQLMRA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.365628162 (498h9m2.236711207s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:03.828213372 (498h9m3.699296417s) [upload=437210]: Next returned nil segment Dec 26 09:38:03.828232628 (498h9m3.699315673s) [upload=437210]: Waiting for error group managing segments... Dec 26 09:38:04.229734875 (498h9m4.10081792s) [noise=true , address=145.79.227.34:4933 , piece_id=7ODKEDJBK3WMG4LHBPVJE2PGMURUMQBHGMA7J2CVNXP6DK5DIL6A, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.251273058 (498h9m4.122356103s) [noise=true , address=145.79.227.7:4322 , piece_id=EKN2EMS5DQCO6QZNIPUSTJIOHB6S5ZXD4NCJCUW2UGRYSGZRUEYQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.252890037 (498h9m4.123973082s) [noise=true , address=145.79.227.40:4081 , piece_id=FQKTTQ4RF3CL4Z7ASHOZFINUSEKUSC3WELUAJHUK5NWQPZZ5NUQA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.359834698 (498h9m4.230917733s) [noise=true , address=64.49.13.140:4508 , piece_id=64FDQH7EMUMDSWRGXSCBD63EGZACEJKZJLXL4IYKSP7RPZHLALNA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.460384842 (498h9m4.331467887s) [noise=true , address=45.91.116.8:4059 , piece_id=VGGWQT6LBVG3I2PGQE2CGNSUETQ6PGPFYWYCGULPF2YYZB4YGUNA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.499297058 (498h9m4.370380103s) [noise=true , address=77.90.31.21:4886 , piece_id=HSVV4VVNNYY6KZR2XPAHYDVPJO6PPBMMRS4JVEOWMZAIVDDNWJGQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.542084071 (498h9m4.413167116s) [noise=true , address=77.90.31.11:4566 , piece_id=L4EZMVP56QYGCJPGWK3FQ4NO5HO7Y4KCRFWKEG2TWATBFEFIGUIA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.582574843 (498h9m4.453657878s) [noise=true , address=45.91.116.10:4100 , piece_id=SIU7MB27XG7QQWM2ZDUADCNCBKJO32TCGJPRUETYKUDZORZK32UA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.844901267 (498h9m4.715984312s) [noise=true , address=77.90.31.16:4756 , piece_id=BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.093586545 (498h9m4.96466959s) [noise=true , address=145.79.227.30:4787 , piece_id=UA4ZRVGGRT7U6MHLC3LTVPA25K6E3XLLLQ2C6766Z24GJHY4WATA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.158083007 (498h9m5.029166052s) [noise=true , address=145.79.227.44:4225 , piece_id=IUI7KC75SGIV7JY2G6KS6I3PAEJ7GOHL6ZJH5F6XQSLSE3YQLMRA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.158420730 (498h9m5.029503776s) [noise=true , address=45.91.116.9:4288 , piece_id=MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.382024590 (498h9m5.253107635s) [noise=true , address=77.90.31.18:4813 , piece_id=LMQL4MOLI2UM2AGA774XJY3FFU2C4OC66TVEBNZKRWQPGZNJ2GYA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.470838497 (498h9m5.341921542s) [noise=true , address=45.91.116.5:4330 , piece_id=YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.470865969 (498h9m5.341949004s) [seg_pos={0 0}, upload=437210]: Segment reached optimal threshold of 14 pieces. Dec 26 09:38:05.471121566 (498h9m5.342204621s) [noise=true , address=77.90.31.15:4725 , piece_id=6HXOF7P376BXRFNGCPIPHQO4ZEREMHUPM373AKMBBVZW27L7AQQA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nstPr2GMJo6pnBmBuDME5QYw3mXgnqLoaTKkcKUhjSbAY8mQk): protocol: expected piece hash; context canceled; EOF Dec 26 09:38:05.471130843 (498h9m5.342213888s) [noise=true , address=77.90.31.15:4725 , piece_id=6HXOF7P376BXRFNGCPIPHQO4ZEREMHUPM373AKMBBVZW27L7AQQA, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471200115 (498h9m5.34228315s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection Dec 26 09:38:05.471216476 (498h9m5.342299511s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471359830 (498h9m5.342442875s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection Dec 26 09:38:05.471368907 (498h9m5.342451942s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471685119 (498h9m5.342768164s) [noise=true , address=77.90.31.13:4649 , piece_id=CSRCWI2DPEM5WZXNTRQI6I72R7YJAB4K2PBPWTBMV7GDUICUFBDQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: read tcp 188.166.225.170:51276->77.90.31.13:4649: use of closed network connection; context canceled Dec 26 09:38:05.471694005 (498h9m5.342777041s) [noise=true , address=77.90.31.13:4649 , piece_id=CSRCWI2DPEM5WZXNTRQI6I72R7YJAB4K2PBPWTBMV7GDUICUFBDQ, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471712161 (498h9m5.342795206s) [seg_pos={0 0}, upload=437210]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 09:38:05.471800789 (498h9m5.342883834s) [upload=437210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:38:05.471805818 (498h9m5.342888853s) [upload=437210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:38:05.775837956 (498h9m5.646921001s) [upload=437210]: Done uploading segments... Dec 26 09:38:05.775863094 (498h9m5.646946129s) [upload=437210]: Upload finished. err: <nil> Duration: 38.769209306s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X....XXX..X.......X.....XX.XX.XXX.....X..X.....X.X...X..XXX] duration: 30.003848832s trace: https://jaeger.dev.storj.io/trace/90411d7e8385af5 FAIL (2025-12-26 09:43:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 26 09:42:53.851110791 (498h13m53.722193826s) [upload=437282]: Got next segment. Inline: false Dec 26 09:42:53.851163452 (498h13m53.722246487s) [upload=437282]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:42:53.851170255 (498h13m53.72225329s) [upload=437282]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:42:54.100350466 (498h13m53.971433501s) [seg_pos={0 0}, upload=437282]: Begin upload segment... Dec 26 09:42:54.100406843 (498h13m53.971489899s) [seg_pos={0 0}, upload=437282]: Done begin upload segment. Dec 26 09:42:54.100470985 (498h13m53.97155402s) [noise=true , address=77.90.31.12:4590 , piece_id=6YNX4QTNS3ZJFIVLX2WF3VGK2EWJOYXCECUFV3VHUB4ADK6E4NCA, seg_pos={0 0}, upload=437282]: Uploading piece... Dec 26 09:42:54.100504088 (498h13m53.971587123s) [noise=true , address=77.90.31.13:4629 , piece_id=YHMTOBPSUEWXEAQ5NFLWMH2TWI4M5GK5PIG25WK5UXZEVGWYUS7Q, seg_pos={0 0}, upload=437282]: Uploading piece... Dec 26 09:42:54.101035851 (498h13m53.972118896s) [noise=true , address=145.79.227.4:4185 , piece_id=7YMSZXEP5L4JCXILA7VXNBBUYZ3WZU5SZKZ2D7OD7WWGU3IGFAEA, seg_pos={0 0}, upload=437282]: Uploading piece... Dec 26 09:42:54.101063012 (498h13m53.972146047s) [noise=true , address=145.79.227.48:4301 , piece_id=3TLJDE6PVO32S2H223ETNKWLIP3JUMKIW746SPZCIMAMU5SNM4JA, seg_pos={0 0}, upload=437282]: Uploading piece... Dec 26 09:42:54.101343928 (498h13m53.972426963s) [noise=true , address=145.79.227.38:4977 , piece_id=VPN5ICWXYWYTP6TMXPOU7QLFQVSGGTURY4UYPBILLR3X26NV3CVA, seg_pos={0 0}, upload=437282]: Uploading piece... Dec 26 09:42:54.101401928 (498h13m53.972484963s) [noise=true , address=145.79.227.36:4519 , piece_id=WOJWRMY54V5XLP4LCXCWEPOABF77PM7AO5IQCNGYXMLRYDGZ4KTA, seg_pos={0 0}, upload=437282]: Uploading piece... Dec 26 09:42:54.101635553 (498h13m53.972718588s) [noise=true , address=45.91.116.9:4261 , piece_id=VVRWAKO7G3ZKVIHQR4O7NJS4MDXB6A5IGK2EU665AX3E7QBBS67A, seg_pos={0 0}, upload=437282]: Uploading piece... Dec 26 09:42:54.101719773 (498h13m53.972802808s) [noise=true , address=77.90.31.17:4773 , piece_id=5VIBBDJD2MSGJRJSB5XPBFVE7NZUAHYZPFJNK73HH74JH4T4XAIA, seg_pos={0 0}, upload=437282]: Uploading piece... Dec 26 09:42:54.102070671 (498h13m53.973153706s) [noise=true , address=145.79.227.3:4151 , piece_id=BBSWJAIP657KX4DAOHULXIGOJXON6YXTKU3J3XRMNO5PKWQCD5NA, seg_pos={0 0}, upload=437282]: Uploading piece... Dec 26 09:42:54.102135615 (498h13m53.97321865s) [noise=true , address=145.79.227.47:4265 , piece_id=2DADJDJF2MC6LOZXPQUW5WBU677IH3QUN5ZNFDBGZR47CJGMBAHA, seg_pos={0 0}, upload=437282]: Uploading piece... Dec 26 09:42:54.102493367 (498h13m53.973576402s) [noise=true , address=145.79.227.45:4231 , piece_id=3UAY6WVZQQ3H6M7D7DBXLTAVL4MTUJ43EXMO2XMC7YGTMRT6J2FQ, seg_pos={0 0}, upload=437282]: Uploading piece... Dec 26 09:42:54.102570553 (498h13m53.973653588s) [noise=true , address=45.91.116.5:4297 , piece_id=UU5FZR4X6JWHX66QZARM3QDRSD74ORV3T2OQKLQFB4Y6N2Q5XNVQ, seg_pos={0 0}, upload=437282]: Uploading piece... Dec 26 09:42:54.102844816 (498h13m53.973927851s) [noise=true , address=45.91.116.10:4081 , piece_id=IGBD44KZZNI7LVQ6JEL2N27DQKIUIQWV24NXWRT5S34Y6OWTKW7Q, seg_pos={0 0}, upload=437282]: Uploading piece... Dec 26 09:42:54.102906132 (498h13m53.973989167s) [noise=true , address=145.79.227.41:4087 , piece_id=HFEG2RDA4FXSOZBUQSFI5VKIDC2IL33OIXMIACNG24C7EGNMN63A, seg_pos={0 0}, upload=437282]: Uploading piece... Dec 26 09:42:54.103105502 (498h13m53.974188547s) [noise=true , address=145.79.227.29:4941 , piece_id=KFVPV5OTDBNCIZN3MEVOB3LA5P5TGWEAPUBRQKAKUW3CT4SSYTBQ, seg_pos={0 0}, upload=437282]: Uploading piece... Dec 26 09:42:54.103355568 (498h13m53.974438613s) [noise=true , address=145.79.227.51:4409 , piece_id=CTWAPRSFZJRENQYML5MIKR3T77RF7FKDR5ZV7BAEFGBMB2FWDLSQ, seg_pos={0 0}, upload=437282]: Uploading piece... Dec 26 09:42:55.808774203 (498h13m55.679857248s) [upload=437282]: Next returned nil segment Dec 26 09:42:55.808808849 (498h13m55.679891914s) [upload=437282]: Waiting for error group managing segments... Dec 26 09:42:56.104724042 (498h13m55.975807077s) [noise=true , address=145.79.227.48:4301 , piece_id=3TLJDE6PVO32S2H223ETNKWLIP3JUMKIW746SPZCIMAMU5SNM4JA, seg_pos={0 0}, upload=437282]: Done uploading piece. err: <nil> Dec 26 09:42:56.105373639 (498h13m55.976456674s) [noise=true , address=145.79.227.45:4231 , piece_id=3UAY6WVZQQ3H6M7D7DBXLTAVL4MTUJ43EXMO2XMC7YGTMRT6J2FQ, seg_pos={0 0}, upload=437282]: Done uploading piece. err: <nil> Dec 26 09:42:56.106968004 (498h13m55.978051039s) [noise=true , address=145.79.227.29:4941 , piece_id=KFVPV5OTDBNCIZN3MEVOB3LA5P5TGWEAPUBRQKAKUW3CT4SSYTBQ, seg_pos={0 0}, upload=437282]: Done uploading piece. err: <nil> Dec 26 09:42:56.113576027 (498h13m55.984659062s) [noise=true , address=145.79.227.38:4977 , piece_id=VPN5ICWXYWYTP6TMXPOU7QLFQVSGGTURY4UYPBILLR3X26NV3CVA, seg_pos={0 0}, upload=437282]: Done uploading piece. err: <nil> Dec 26 09:42:56.120207083 (498h13m55.991290119s) [noise=true , address=145.79.227.47:4265 , piece_id=2DADJDJF2MC6LOZXPQUW5WBU677IH3QUN5ZNFDBGZR47CJGMBAHA, seg_pos={0 0}, upload=437282]: Done uploading piece. err: <nil> Dec 26 09:42:56.121661373 (498h13m55.992744408s) [noise=true , address=145.79.227.4:4185 , piece_id=7YMSZXEP5L4JCXILA7VXNBBUYZ3WZU5SZKZ2D7OD7WWGU3IGFAEA, seg_pos={0 0}, upload=437282]: Done uploading piece. err: <nil> Dec 26 09:42:56.126407700 (498h13m55.997490745s) [noise=true , address=145.79.227.41:4087 , piece_id=HFEG2RDA4FXSOZBUQSFI5VKIDC2IL33OIXMIACNG24C7EGNMN63A, seg_pos={0 0}, upload=437282]: Done uploading piece. err: <nil> Dec 26 09:42:56.130581588 (498h13m56.001664623s) [noise=true , address=145.79.227.36:4519 , piece_id=WOJWRMY54V5XLP4LCXCWEPOABF77PM7AO5IQCNGYXMLRYDGZ4KTA, seg_pos={0 0}, upload=437282]: Done uploading piece. err: <nil> Dec 26 09:42:56.137375326 (498h13m56.008458371s) [noise=true , address=145.79.227.51:4409 , piece_id=CTWAPRSFZJRENQYML5MIKR3T77RF7FKDR5ZV7BAEFGBMB2FWDLSQ, seg_pos={0 0}, upload=437282]: Done uploading piece. err: <nil> Dec 26 09:42:56.212178082 (498h13m56.083261127s) [noise=true , address=145.79.227.3:4151 , piece_id=BBSWJAIP657KX4DAOHULXIGOJXON6YXTKU3J3XRMNO5PKWQCD5NA, seg_pos={0 0}, upload=437282]: Done uploading piece. err: <nil> Dec 26 09:42:56.381866788 (498h13m56.252949833s) [noise=true , address=77.90.31.17:4773 , piece_id=5VIBBDJD2MSGJRJSB5XPBFVE7NZUAHYZPFJNK73HH74JH4T4XAIA, seg_pos={0 0}, upload=437282]: Done uploading piece. err: <nil> Dec 26 09:42:56.623939155 (498h13m56.4950222s) [noise=true , address=77.90.31.12:4590 , piece_id=6YNX4QTNS3ZJFIVLX2WF3VGK2EWJOYXCECUFV3VHUB4ADK6E4NCA, seg_pos={0 0}, upload=437282]: Done uploading piece. err: <nil> Dec 26 09:42:56.694572543 (498h13m56.565655578s) [noise=true , address=45.91.116.9:4261 , piece_id=VVRWAKO7G3ZKVIHQR4O7NJS4MDXB6A5IGK2EU665AX3E7QBBS67A, seg_pos={0 0}, upload=437282]: Done uploading piece. err: <nil> Dec 26 09:42:56.694591750 (498h13m56.565674785s) [seg_pos={0 0}, upload=437282]: Segment reached optimal threshold of 13 pieces. Dec 26 09:42:56.694810296 (498h13m56.565893331s) [noise=true , address=45.91.116.5:4297 , piece_id=UU5FZR4X6JWHX66QZARM3QDRSD74ORV3T2OQKLQFB4Y6N2Q5XNVQ, seg_pos={0 0}, upload=437282]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 09:42:56.694816557 (498h13m56.565899592s) [noise=true , address=45.91.116.5:4297 , piece_id=UU5FZR4X6JWHX66QZARM3QDRSD74ORV3T2OQKLQFB4Y6N2Q5XNVQ, seg_pos={0 0}, upload=437282]: Long tail cancellation due to optimal threshold Dec 26 09:42:56.695082123 (498h13m56.566165159s) [noise=true , address=77.90.31.13:4629 , piece_id=YHMTOBPSUEWXEAQ5NFLWMH2TWI4M5GK5PIG25WK5UXZEVGWYUS7Q, seg_pos={0 0}, upload=437282]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:60052->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:60052->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60052->77.90.31.13:4629: use of closed network connection Dec 26 09:42:56.695090199 (498h13m56.566173234s) [noise=true , address=77.90.31.13:4629 , piece_id=YHMTOBPSUEWXEAQ5NFLWMH2TWI4M5GK5PIG25WK5UXZEVGWYUS7Q, seg_pos={0 0}, upload=437282]: Long tail cancellation due to optimal threshold Dec 26 09:42:56.695178257 (498h13m56.566261302s) [noise=true , address=45.91.116.10:4081 , piece_id=IGBD44KZZNI7LVQ6JEL2N27DQKIUIQWV24NXWRT5S34Y6OWTKW7Q, seg_pos={0 0}, upload=437282]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48110->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48110->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48110->45.91.116.10:4081: use of closed network connection Dec 26 09:42:56.695183597 (498h13m56.566266632s) [noise=true , address=45.91.116.10:4081 , piece_id=IGBD44KZZNI7LVQ6JEL2N27DQKIUIQWV24NXWRT5S34Y6OWTKW7Q, seg_pos={0 0}, upload=437282]: Long tail cancellation due to optimal threshold Dec 26 09:42:56.695205468 (498h13m56.566288503s) [seg_pos={0 0}, upload=437282]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:42:56.695285571 (498h13m56.566368606s) [upload=437282]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:42:56.695291923 (498h13m56.566374968s) [upload=437282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:42:56.993559765 (498h13m56.8646428s) [upload=437282]: Done uploading segments... Dec 26 09:42:56.993578841 (498h13m56.864661876s) [upload=437282]: Upload finished. err: <nil> Duration: 30.003870965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 32.131935158s trace: https://jaeger.dev.storj.io/trace/667977214d97ec8c FAIL (2025-12-26 09:43:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, Piece ID: YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA) context deadline exceeded; piecestore: (Node ID: 1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, Piece ID: MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A) context deadline exceeded ======LOGS====== true , address=77.90.31.11:4566 , piece_id=L4EZMVP56QYGCJPGWK3FQ4NO5HO7Y4KCRFWKEG2TWATBFEFIGUIA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.357693223 (498h9m2.228776258s) [noise=true , address=77.90.31.21:4886 , piece_id=HSVV4VVNNYY6KZR2XPAHYDVPJO6PPBMMRS4JVEOWMZAIVDDNWJGQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.358418583 (498h9m2.229501628s) [noise=true , address=64.49.13.140:4508 , piece_id=64FDQH7EMUMDSWRGXSCBD63EGZACEJKZJLXL4IYKSP7RPZHLALNA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.358935137 (498h9m2.230018182s) [noise=true , address=77.90.31.16:4756 , piece_id=BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359491216 (498h9m2.230574261s) [noise=true , address=45.91.116.9:4288 , piece_id=MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359953967 (498h9m2.231037013s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359968154 (498h9m2.231051199s) [noise=true , address=77.90.31.18:4813 , piece_id=LMQL4MOLI2UM2AGA774XJY3FFU2C4OC66TVEBNZKRWQPGZNJ2GYA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.360429653 (498h9m2.231512698s) [noise=true , address=145.79.227.40:4081 , piece_id=FQKTTQ4RF3CL4Z7ASHOZFINUSEKUSC3WELUAJHUK5NWQPZZ5NUQA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.360518873 (498h9m2.231601918s) [noise=true , address=145.79.227.7:4322 , piece_id=EKN2EMS5DQCO6QZNIPUSTJIOHB6S5ZXD4NCJCUW2UGRYSGZRUEYQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361114697 (498h9m2.232197742s) [noise=true , address=45.91.116.8:4059 , piece_id=VGGWQT6LBVG3I2PGQE2CGNSUETQ6PGPFYWYCGULPF2YYZB4YGUNA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361136940 (498h9m2.232219985s) [noise=true , address=45.91.116.5:4330 , piece_id=YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361631942 (498h9m2.232714988s) [noise=true , address=45.91.116.10:4100 , piece_id=SIU7MB27XG7QQWM2ZDUADCNCBKJO32TCGJPRUETYKUDZORZK32UA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361832193 (498h9m2.232915238s) [noise=true , address=145.79.227.34:4933 , piece_id=7ODKEDJBK3WMG4LHBPVJE2PGMURUMQBHGMA7J2CVNXP6DK5DIL6A, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.362940474 (498h9m2.234023529s) [noise=true , address=145.79.227.44:4225 , piece_id=IUI7KC75SGIV7JY2G6KS6I3PAEJ7GOHL6ZJH5F6XQSLSE3YQLMRA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.365628162 (498h9m2.236711207s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:03.828213372 (498h9m3.699296417s) [upload=437210]: Next returned nil segment Dec 26 09:38:03.828232628 (498h9m3.699315673s) [upload=437210]: Waiting for error group managing segments... Dec 26 09:38:04.229734875 (498h9m4.10081792s) [noise=true , address=145.79.227.34:4933 , piece_id=7ODKEDJBK3WMG4LHBPVJE2PGMURUMQBHGMA7J2CVNXP6DK5DIL6A, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.251273058 (498h9m4.122356103s) [noise=true , address=145.79.227.7:4322 , piece_id=EKN2EMS5DQCO6QZNIPUSTJIOHB6S5ZXD4NCJCUW2UGRYSGZRUEYQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.252890037 (498h9m4.123973082s) [noise=true , address=145.79.227.40:4081 , piece_id=FQKTTQ4RF3CL4Z7ASHOZFINUSEKUSC3WELUAJHUK5NWQPZZ5NUQA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.359834698 (498h9m4.230917733s) [noise=true , address=64.49.13.140:4508 , piece_id=64FDQH7EMUMDSWRGXSCBD63EGZACEJKZJLXL4IYKSP7RPZHLALNA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.460384842 (498h9m4.331467887s) [noise=true , address=45.91.116.8:4059 , piece_id=VGGWQT6LBVG3I2PGQE2CGNSUETQ6PGPFYWYCGULPF2YYZB4YGUNA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.499297058 (498h9m4.370380103s) [noise=true , address=77.90.31.21:4886 , piece_id=HSVV4VVNNYY6KZR2XPAHYDVPJO6PPBMMRS4JVEOWMZAIVDDNWJGQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.542084071 (498h9m4.413167116s) [noise=true , address=77.90.31.11:4566 , piece_id=L4EZMVP56QYGCJPGWK3FQ4NO5HO7Y4KCRFWKEG2TWATBFEFIGUIA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.582574843 (498h9m4.453657878s) [noise=true , address=45.91.116.10:4100 , piece_id=SIU7MB27XG7QQWM2ZDUADCNCBKJO32TCGJPRUETYKUDZORZK32UA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.844901267 (498h9m4.715984312s) [noise=true , address=77.90.31.16:4756 , piece_id=BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.093586545 (498h9m4.96466959s) [noise=true , address=145.79.227.30:4787 , piece_id=UA4ZRVGGRT7U6MHLC3LTVPA25K6E3XLLLQ2C6766Z24GJHY4WATA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.158083007 (498h9m5.029166052s) [noise=true , address=145.79.227.44:4225 , piece_id=IUI7KC75SGIV7JY2G6KS6I3PAEJ7GOHL6ZJH5F6XQSLSE3YQLMRA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.158420730 (498h9m5.029503776s) [noise=true , address=45.91.116.9:4288 , piece_id=MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.382024590 (498h9m5.253107635s) [noise=true , address=77.90.31.18:4813 , piece_id=LMQL4MOLI2UM2AGA774XJY3FFU2C4OC66TVEBNZKRWQPGZNJ2GYA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.470838497 (498h9m5.341921542s) [noise=true , address=45.91.116.5:4330 , piece_id=YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.470865969 (498h9m5.341949004s) [seg_pos={0 0}, upload=437210]: Segment reached optimal threshold of 14 pieces. Dec 26 09:38:05.471121566 (498h9m5.342204621s) [noise=true , address=77.90.31.15:4725 , piece_id=6HXOF7P376BXRFNGCPIPHQO4ZEREMHUPM373AKMBBVZW27L7AQQA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nstPr2GMJo6pnBmBuDME5QYw3mXgnqLoaTKkcKUhjSbAY8mQk): protocol: expected piece hash; context canceled; EOF Dec 26 09:38:05.471130843 (498h9m5.342213888s) [noise=true , address=77.90.31.15:4725 , piece_id=6HXOF7P376BXRFNGCPIPHQO4ZEREMHUPM373AKMBBVZW27L7AQQA, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471200115 (498h9m5.34228315s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection Dec 26 09:38:05.471216476 (498h9m5.342299511s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471359830 (498h9m5.342442875s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection Dec 26 09:38:05.471368907 (498h9m5.342451942s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471685119 (498h9m5.342768164s) [noise=true , address=77.90.31.13:4649 , piece_id=CSRCWI2DPEM5WZXNTRQI6I72R7YJAB4K2PBPWTBMV7GDUICUFBDQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: read tcp 188.166.225.170:51276->77.90.31.13:4649: use of closed network connection; context canceled Dec 26 09:38:05.471694005 (498h9m5.342777041s) [noise=true , address=77.90.31.13:4649 , piece_id=CSRCWI2DPEM5WZXNTRQI6I72R7YJAB4K2PBPWTBMV7GDUICUFBDQ, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471712161 (498h9m5.342795206s) [seg_pos={0 0}, upload=437210]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 09:38:05.471800789 (498h9m5.342883834s) [upload=437210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:38:05.471805818 (498h9m5.342888853s) [upload=437210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:38:05.775837956 (498h9m5.646921001s) [upload=437210]: Done uploading segments... Dec 26 09:38:05.775863094 (498h9m5.646946129s) [upload=437210]: Upload finished. err: <nil> Duration: 32.131960648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 41.274428638s trace: https://jaeger.dev.storj.io/trace/667977214d97ec8c FAIL (2025-12-26 09:42:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, Piece ID: YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA) context deadline exceeded; piecestore: (Node ID: 1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, Piece ID: MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A) context deadline exceeded ======LOGS====== true , address=77.90.31.11:4566 , piece_id=L4EZMVP56QYGCJPGWK3FQ4NO5HO7Y4KCRFWKEG2TWATBFEFIGUIA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.357693223 (498h9m2.228776258s) [noise=true , address=77.90.31.21:4886 , piece_id=HSVV4VVNNYY6KZR2XPAHYDVPJO6PPBMMRS4JVEOWMZAIVDDNWJGQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.358418583 (498h9m2.229501628s) [noise=true , address=64.49.13.140:4508 , piece_id=64FDQH7EMUMDSWRGXSCBD63EGZACEJKZJLXL4IYKSP7RPZHLALNA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.358935137 (498h9m2.230018182s) [noise=true , address=77.90.31.16:4756 , piece_id=BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359491216 (498h9m2.230574261s) [noise=true , address=45.91.116.9:4288 , piece_id=MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359953967 (498h9m2.231037013s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359968154 (498h9m2.231051199s) [noise=true , address=77.90.31.18:4813 , piece_id=LMQL4MOLI2UM2AGA774XJY3FFU2C4OC66TVEBNZKRWQPGZNJ2GYA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.360429653 (498h9m2.231512698s) [noise=true , address=145.79.227.40:4081 , piece_id=FQKTTQ4RF3CL4Z7ASHOZFINUSEKUSC3WELUAJHUK5NWQPZZ5NUQA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.360518873 (498h9m2.231601918s) [noise=true , address=145.79.227.7:4322 , piece_id=EKN2EMS5DQCO6QZNIPUSTJIOHB6S5ZXD4NCJCUW2UGRYSGZRUEYQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361114697 (498h9m2.232197742s) [noise=true , address=45.91.116.8:4059 , piece_id=VGGWQT6LBVG3I2PGQE2CGNSUETQ6PGPFYWYCGULPF2YYZB4YGUNA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361136940 (498h9m2.232219985s) [noise=true , address=45.91.116.5:4330 , piece_id=YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361631942 (498h9m2.232714988s) [noise=true , address=45.91.116.10:4100 , piece_id=SIU7MB27XG7QQWM2ZDUADCNCBKJO32TCGJPRUETYKUDZORZK32UA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361832193 (498h9m2.232915238s) [noise=true , address=145.79.227.34:4933 , piece_id=7ODKEDJBK3WMG4LHBPVJE2PGMURUMQBHGMA7J2CVNXP6DK5DIL6A, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.362940474 (498h9m2.234023529s) [noise=true , address=145.79.227.44:4225 , piece_id=IUI7KC75SGIV7JY2G6KS6I3PAEJ7GOHL6ZJH5F6XQSLSE3YQLMRA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.365628162 (498h9m2.236711207s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:03.828213372 (498h9m3.699296417s) [upload=437210]: Next returned nil segment Dec 26 09:38:03.828232628 (498h9m3.699315673s) [upload=437210]: Waiting for error group managing segments... Dec 26 09:38:04.229734875 (498h9m4.10081792s) [noise=true , address=145.79.227.34:4933 , piece_id=7ODKEDJBK3WMG4LHBPVJE2PGMURUMQBHGMA7J2CVNXP6DK5DIL6A, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.251273058 (498h9m4.122356103s) [noise=true , address=145.79.227.7:4322 , piece_id=EKN2EMS5DQCO6QZNIPUSTJIOHB6S5ZXD4NCJCUW2UGRYSGZRUEYQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.252890037 (498h9m4.123973082s) [noise=true , address=145.79.227.40:4081 , piece_id=FQKTTQ4RF3CL4Z7ASHOZFINUSEKUSC3WELUAJHUK5NWQPZZ5NUQA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.359834698 (498h9m4.230917733s) [noise=true , address=64.49.13.140:4508 , piece_id=64FDQH7EMUMDSWRGXSCBD63EGZACEJKZJLXL4IYKSP7RPZHLALNA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.460384842 (498h9m4.331467887s) [noise=true , address=45.91.116.8:4059 , piece_id=VGGWQT6LBVG3I2PGQE2CGNSUETQ6PGPFYWYCGULPF2YYZB4YGUNA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.499297058 (498h9m4.370380103s) [noise=true , address=77.90.31.21:4886 , piece_id=HSVV4VVNNYY6KZR2XPAHYDVPJO6PPBMMRS4JVEOWMZAIVDDNWJGQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.542084071 (498h9m4.413167116s) [noise=true , address=77.90.31.11:4566 , piece_id=L4EZMVP56QYGCJPGWK3FQ4NO5HO7Y4KCRFWKEG2TWATBFEFIGUIA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.582574843 (498h9m4.453657878s) [noise=true , address=45.91.116.10:4100 , piece_id=SIU7MB27XG7QQWM2ZDUADCNCBKJO32TCGJPRUETYKUDZORZK32UA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.844901267 (498h9m4.715984312s) [noise=true , address=77.90.31.16:4756 , piece_id=BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.093586545 (498h9m4.96466959s) [noise=true , address=145.79.227.30:4787 , piece_id=UA4ZRVGGRT7U6MHLC3LTVPA25K6E3XLLLQ2C6766Z24GJHY4WATA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.158083007 (498h9m5.029166052s) [noise=true , address=145.79.227.44:4225 , piece_id=IUI7KC75SGIV7JY2G6KS6I3PAEJ7GOHL6ZJH5F6XQSLSE3YQLMRA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.158420730 (498h9m5.029503776s) [noise=true , address=45.91.116.9:4288 , piece_id=MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.382024590 (498h9m5.253107635s) [noise=true , address=77.90.31.18:4813 , piece_id=LMQL4MOLI2UM2AGA774XJY3FFU2C4OC66TVEBNZKRWQPGZNJ2GYA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.470838497 (498h9m5.341921542s) [noise=true , address=45.91.116.5:4330 , piece_id=YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.470865969 (498h9m5.341949004s) [seg_pos={0 0}, upload=437210]: Segment reached optimal threshold of 14 pieces. Dec 26 09:38:05.471121566 (498h9m5.342204621s) [noise=true , address=77.90.31.15:4725 , piece_id=6HXOF7P376BXRFNGCPIPHQO4ZEREMHUPM373AKMBBVZW27L7AQQA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nstPr2GMJo6pnBmBuDME5QYw3mXgnqLoaTKkcKUhjSbAY8mQk): protocol: expected piece hash; context canceled; EOF Dec 26 09:38:05.471130843 (498h9m5.342213888s) [noise=true , address=77.90.31.15:4725 , piece_id=6HXOF7P376BXRFNGCPIPHQO4ZEREMHUPM373AKMBBVZW27L7AQQA, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471200115 (498h9m5.34228315s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection Dec 26 09:38:05.471216476 (498h9m5.342299511s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471359830 (498h9m5.342442875s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection Dec 26 09:38:05.471368907 (498h9m5.342451942s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471685119 (498h9m5.342768164s) [noise=true , address=77.90.31.13:4649 , piece_id=CSRCWI2DPEM5WZXNTRQI6I72R7YJAB4K2PBPWTBMV7GDUICUFBDQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: read tcp 188.166.225.170:51276->77.90.31.13:4649: use of closed network connection; context canceled Dec 26 09:38:05.471694005 (498h9m5.342777041s) [noise=true , address=77.90.31.13:4649 , piece_id=CSRCWI2DPEM5WZXNTRQI6I72R7YJAB4K2PBPWTBMV7GDUICUFBDQ, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471712161 (498h9m5.342795206s) [seg_pos={0 0}, upload=437210]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 09:38:05.471800789 (498h9m5.342883834s) [upload=437210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:38:05.471805818 (498h9m5.342888853s) [upload=437210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:38:05.775837956 (498h9m5.646921001s) [upload=437210]: Done uploading segments... Dec 26 09:38:05.775863094 (498h9m5.646946129s) [upload=437210]: Upload finished. err: <nil> Duration: 41.274455539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X....XXX..X.......X.....XX.XX.XXX.....X..X.....X.X...X..XX] duration: 30.003341616s trace: https://jaeger.dev.storj.io/trace/6701d77f0fcabf64 FAIL (2025-12-26 09:42:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 6NKUZQLJHGT76AICUAFYYZP5GNG6ZQTLMEFSIGGNV45T5P52ZJKQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: Y35PXCTVKVHLGQ7CGDBQ5RPGW4ZZ2OMGYF7MMSLCDPPMXB5XVQIQ) context deadline exceeded ======LOGS====== Dec 26 09:41:47.406740696 (498h12m47.277823741s) [upload=437267]: Starting upload Dec 26 09:41:47.406902615 (498h12m47.27798566s) [upload=437267]: Done starting upload Dec 26 09:41:47.407021050 (498h12m47.278104095s) [upload=437267]: Uploading segments... Dec 26 09:41:47.407141439 (498h12m47.278224484s) [upload=437267]: Got next segment. Inline: false Dec 26 09:41:47.407180383 (498h12m47.278263418s) [upload=437267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:41:47.407185604 (498h12m47.278268639s) [upload=437267]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:41:47.650756584 (498h12m47.521839629s) [seg_pos={0 0}, upload=437267]: Begin upload segment... Dec 26 09:41:47.650819483 (498h12m47.521902528s) [seg_pos={0 0}, upload=437267]: Done begin upload segment. Dec 26 09:41:47.650838389 (498h12m47.521921424s) [noise=true , address=145.79.227.13:4803 , piece_id=XKPOQYA4WEGW7ATDKEYD4A2YQDE73ZGVUMYTZBQI337ZNXNPT2ZQ, seg_pos={0 0}, upload=437267]: Uploading piece... Dec 26 09:41:47.650911559 (498h12m47.521994604s) [noise=true , address=145.79.227.32:4835 , piece_id=WRDQNYDOAABGEFT72T4SDLSBBW2DCJK2PI7UTNK4WAAYOI7UKQKA, seg_pos={0 0}, upload=437267]: Uploading piece... Dec 26 09:41:47.651373869 (498h12m47.522456904s) [noise=true , address=45.91.116.14:4190 , piece_id=NIHWMERVC34EGUBEDYA7WB6LNWHWJM5YQFPSX3MCL5J4XV5NDM5Q, seg_pos={0 0}, upload=437267]: Uploading piece... Dec 26 09:41:47.651786484 (498h12m47.52286953s) [noise=true , address=145.79.227.35:4482 , piece_id=VQM7LNOJCLRUU4FSFP4433WXZXESXPFC5WK32PXQNNLQ5DPSOLIQ, seg_pos={0 0}, upload=437267]: Uploading piece... Dec 26 09:41:47.652228687 (498h12m47.523311722s) [noise=true , address=145.79.227.49:4339 , piece_id=3CO44NMKIBUB3MB7G6GHZAZWOJ7Y3Y5WZ56HVSNFZP5IERZH3GRQ, seg_pos={0 0}, upload=437267]: Uploading piece... Dec 26 09:41:47.652671430 (498h12m47.523754475s) [noise=true , address=145.79.227.52:4445 , piece_id=X5FI4FH3CWUT34RDRHAJWDWRLRGO4X3EQVLHRZAEST46OHE2FAYQ, seg_pos={0 0}, upload=437267]: Uploading piece... Dec 26 09:41:47.653219212 (498h12m47.524302248s) [noise=true , address=145.79.227.48:4303 , piece_id=UNB26BXW4UBZRUL6XUAGJVK3U7TJRCA7SCHIBP3VO6OZIU77ZU6Q, seg_pos={0 0}, upload=437267]: Uploading piece... Dec 26 09:41:47.654865047 (498h12m47.525948082s) [noise=true , address=77.90.31.21:4878 , piece_id=YKZGV6M63IA7EYUOHQY3R4USMPEGU5VFGUW2Q5PE4KEEFU2BGWNA, seg_pos={0 0}, upload=437267]: Uploading piece... Dec 26 09:41:47.657118207 (498h12m47.528201252s) [noise=true , address=45.91.116.7:4011 , piece_id=ICA7E5DVMV5GNVTOIE3M5GTOTJPF6APECUECBUARXA2PCJOQKZYQ, seg_pos={0 0}, upload=437267]: Uploading piece... Dec 26 09:41:47.659330770 (498h12m47.530413815s) [noise=true , address=45.91.116.13:4154 , piece_id=6NKUZQLJHGT76AICUAFYYZP5GNG6ZQTLMEFSIGGNV45T5P52ZJKQ, seg_pos={0 0}, upload=437267]: Uploading piece... Dec 26 09:41:47.661226710 (498h12m47.532309755s) [noise=true , address=145.79.227.42:4122 , piece_id=G5YHHHEGMPFMOZKMA4OOAWUE34V4IS35TPIRYJ3U25QCMLGTS2QQ, seg_pos={0 0}, upload=437267]: Uploading piece... Dec 26 09:41:47.663115287 (498h12m47.534198322s) [noise=true , address=145.79.227.20:4437 , piece_id=IYFDCKNPSOYUXW257LREHNABN3F7OBA5MKVLTUCSHTJYJZBZXWWQ, seg_pos={0 0}, upload=437267]: Uploading piece... Dec 26 09:41:47.664570628 (498h12m47.535653663s) [noise=true , address=145.79.227.37:4566 , piece_id=DHAATBNRWTVD3JCBOBMVO2O7UM45VHUBB6ZWF666DSJXKQLQ656Q, seg_pos={0 0}, upload=437267]: Uploading piece... Dec 26 09:41:47.666205932 (498h12m47.537288967s) [noise=true , address=77.90.31.17:4770 , piece_id=RLKTW6WIHQYFYDAQ3UGQ7YGWGRUOBWZM6PFCMCK7IE7ZQQTJHC5Q, seg_pos={0 0}, upload=437267]: Uploading piece... Dec 26 09:41:47.668425348 (498h12m47.539508383s) [noise=true , address=45.91.116.11:4118 , piece_id=Y35PXCTVKVHLGQ7CGDBQ5RPGW4ZZ2OMGYF7MMSLCDPPMXB5XVQIQ, seg_pos={0 0}, upload=437267]: Uploading piece... Dec 26 09:41:47.669686048 (498h12m47.540769083s) [noise=true , address=145.79.227.11:4729 , piece_id=6TC26TYJB6GIDKRJUA3NP45VL7LIWA33RHNHPANIG7T6CSAOIX7Q, seg_pos={0 0}, upload=437267]: Uploading piece... Dec 26 09:41:49.373279017 (498h12m49.244362052s) [upload=437267]: Next returned nil segment Dec 26 09:41:49.373293635 (498h12m49.24437668s) [upload=437267]: Waiting for error group managing segments... Dec 26 09:41:49.656412431 (498h12m49.527495466s) [noise=true , address=145.79.227.13:4803 , piece_id=XKPOQYA4WEGW7ATDKEYD4A2YQDE73ZGVUMYTZBQI337ZNXNPT2ZQ, seg_pos={0 0}, upload=437267]: Done uploading piece. err: <nil> Dec 26 09:41:49.658878387 (498h12m49.529961422s) [noise=true , address=145.79.227.48:4303 , piece_id=UNB26BXW4UBZRUL6XUAGJVK3U7TJRCA7SCHIBP3VO6OZIU77ZU6Q, seg_pos={0 0}, upload=437267]: Done uploading piece. err: <nil> Dec 26 09:41:49.668946017 (498h12m49.540029052s) [noise=true , address=145.79.227.32:4835 , piece_id=WRDQNYDOAABGEFT72T4SDLSBBW2DCJK2PI7UTNK4WAAYOI7UKQKA, seg_pos={0 0}, upload=437267]: Done uploading piece. err: <nil> Dec 26 09:41:49.673427651 (498h12m49.544510686s) [noise=true , address=145.79.227.11:4729 , piece_id=6TC26TYJB6GIDKRJUA3NP45VL7LIWA33RHNHPANIG7T6CSAOIX7Q, seg_pos={0 0}, upload=437267]: Done uploading piece. err: <nil> Dec 26 09:41:49.674471419 (498h12m49.545554454s) [noise=true , address=145.79.227.20:4437 , piece_id=IYFDCKNPSOYUXW257LREHNABN3F7OBA5MKVLTUCSHTJYJZBZXWWQ, seg_pos={0 0}, upload=437267]: Done uploading piece. err: <nil> Dec 26 09:41:49.675740384 (498h12m49.54682343s) [noise=true , address=145.79.227.52:4445 , piece_id=X5FI4FH3CWUT34RDRHAJWDWRLRGO4X3EQVLHRZAEST46OHE2FAYQ, seg_pos={0 0}, upload=437267]: Done uploading piece. err: <nil> Dec 26 09:41:49.687753039 (498h12m49.558836074s) [noise=true , address=145.79.227.49:4339 , piece_id=3CO44NMKIBUB3MB7G6GHZAZWOJ7Y3Y5WZ56HVSNFZP5IERZH3GRQ, seg_pos={0 0}, upload=437267]: Done uploading piece. err: <nil> Dec 26 09:41:49.698036430 (498h12m49.569119476s) [noise=true , address=145.79.227.37:4566 , piece_id=DHAATBNRWTVD3JCBOBMVO2O7UM45VHUBB6ZWF666DSJXKQLQ656Q, seg_pos={0 0}, upload=437267]: Done uploading piece. err: <nil> Dec 26 09:41:49.700818959 (498h12m49.571901994s) [noise=true , address=145.79.227.35:4482 , piece_id=VQM7LNOJCLRUU4FSFP4433WXZXESXPFC5WK32PXQNNLQ5DPSOLIQ, seg_pos={0 0}, upload=437267]: Done uploading piece. err: <nil> Dec 26 09:41:49.702709809 (498h12m49.573792844s) [noise=true , address=145.79.227.42:4122 , piece_id=G5YHHHEGMPFMOZKMA4OOAWUE34V4IS35TPIRYJ3U25QCMLGTS2QQ, seg_pos={0 0}, upload=437267]: Done uploading piece. err: <nil> Dec 26 09:41:49.891393654 (498h12m49.7624767s) [noise=true , address=45.91.116.11:4118 , piece_id=Y35PXCTVKVHLGQ7CGDBQ5RPGW4ZZ2OMGYF7MMSLCDPPMXB5XVQIQ, seg_pos={0 0}, upload=437267]: Done uploading piece. err: <nil> Dec 26 09:41:51.411221664 (498h12m51.282304709s) [noise=true , address=77.90.31.21:4878 , piece_id=YKZGV6M63IA7EYUOHQY3R4USMPEGU5VFGUW2Q5PE4KEEFU2BGWNA, seg_pos={0 0}, upload=437267]: Done uploading piece. err: <nil> Dec 26 09:41:51.773922688 (498h12m51.645005733s) [noise=true , address=45.91.116.13:4154 , piece_id=6NKUZQLJHGT76AICUAFYYZP5GNG6ZQTLMEFSIGGNV45T5P52ZJKQ, seg_pos={0 0}, upload=437267]: Done uploading piece. err: <nil> Dec 26 09:41:51.773946953 (498h12m51.645029998s) [seg_pos={0 0}, upload=437267]: Segment reached optimal threshold of 13 pieces. Dec 26 09:41:51.774267825 (498h12m51.64535088s) [noise=true , address=77.90.31.17:4770 , piece_id=RLKTW6WIHQYFYDAQ3UGQ7YGWGRUOBWZM6PFCMCK7IE7ZQQTJHC5Q, seg_pos={0 0}, upload=437267]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 09:41:51.774277924 (498h12m51.645360969s) [noise=true , address=77.90.31.17:4770 , piece_id=RLKTW6WIHQYFYDAQ3UGQ7YGWGRUOBWZM6PFCMCK7IE7ZQQTJHC5Q, seg_pos={0 0}, upload=437267]: Long tail cancellation due to optimal threshold Dec 26 09:41:51.774466783 (498h12m51.645549838s) [noise=true , address=45.91.116.7:4011 , piece_id=ICA7E5DVMV5GNVTOIE3M5GTOTJPF6APECUECBUARXA2PCJOQKZYQ, seg_pos={0 0}, upload=437267]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 09:41:51.774475951 (498h12m51.645559007s) [noise=true , address=45.91.116.7:4011 , piece_id=ICA7E5DVMV5GNVTOIE3M5GTOTJPF6APECUECBUARXA2PCJOQKZYQ, seg_pos={0 0}, upload=437267]: Long tail cancellation due to optimal threshold Dec 26 09:41:51.774528110 (498h12m51.645611155s) [noise=true , address=45.91.116.14:4190 , piece_id=NIHWMERVC34EGUBEDYA7WB6LNWHWJM5YQFPSX3MCL5J4XV5NDM5Q, seg_pos={0 0}, upload=437267]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:48168->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 09:41:51.774544933 (498h12m51.645627968s) [noise=true , address=45.91.116.14:4190 , piece_id=NIHWMERVC34EGUBEDYA7WB6LNWHWJM5YQFPSX3MCL5J4XV5NDM5Q, seg_pos={0 0}, upload=437267]: Long tail cancellation due to optimal threshold Dec 26 09:41:51.774567465 (498h12m51.6456505s) [seg_pos={0 0}, upload=437267]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:41:51.774681221 (498h12m51.645764266s) [upload=437267]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:41:51.774689538 (498h12m51.645772583s) [upload=437267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:41:52.070592107 (498h12m51.941675142s) [upload=437267]: Done uploading segments... Dec 26 09:41:52.070614299 (498h12m51.941697344s) [upload=437267]: Upload finished. err: <nil> Duration: 30.003363046s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 50.576431537s trace: https://jaeger.dev.storj.io/trace/667977214d97ec8c FAIL (2025-12-26 09:41:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, Piece ID: YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA) context deadline exceeded; piecestore: (Node ID: 1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, Piece ID: MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A) context deadline exceeded ======LOGS====== true , address=77.90.31.11:4566 , piece_id=L4EZMVP56QYGCJPGWK3FQ4NO5HO7Y4KCRFWKEG2TWATBFEFIGUIA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.357693223 (498h9m2.228776258s) [noise=true , address=77.90.31.21:4886 , piece_id=HSVV4VVNNYY6KZR2XPAHYDVPJO6PPBMMRS4JVEOWMZAIVDDNWJGQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.358418583 (498h9m2.229501628s) [noise=true , address=64.49.13.140:4508 , piece_id=64FDQH7EMUMDSWRGXSCBD63EGZACEJKZJLXL4IYKSP7RPZHLALNA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.358935137 (498h9m2.230018182s) [noise=true , address=77.90.31.16:4756 , piece_id=BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359491216 (498h9m2.230574261s) [noise=true , address=45.91.116.9:4288 , piece_id=MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359953967 (498h9m2.231037013s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359968154 (498h9m2.231051199s) [noise=true , address=77.90.31.18:4813 , piece_id=LMQL4MOLI2UM2AGA774XJY3FFU2C4OC66TVEBNZKRWQPGZNJ2GYA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.360429653 (498h9m2.231512698s) [noise=true , address=145.79.227.40:4081 , piece_id=FQKTTQ4RF3CL4Z7ASHOZFINUSEKUSC3WELUAJHUK5NWQPZZ5NUQA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.360518873 (498h9m2.231601918s) [noise=true , address=145.79.227.7:4322 , piece_id=EKN2EMS5DQCO6QZNIPUSTJIOHB6S5ZXD4NCJCUW2UGRYSGZRUEYQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361114697 (498h9m2.232197742s) [noise=true , address=45.91.116.8:4059 , piece_id=VGGWQT6LBVG3I2PGQE2CGNSUETQ6PGPFYWYCGULPF2YYZB4YGUNA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361136940 (498h9m2.232219985s) [noise=true , address=45.91.116.5:4330 , piece_id=YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361631942 (498h9m2.232714988s) [noise=true , address=45.91.116.10:4100 , piece_id=SIU7MB27XG7QQWM2ZDUADCNCBKJO32TCGJPRUETYKUDZORZK32UA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361832193 (498h9m2.232915238s) [noise=true , address=145.79.227.34:4933 , piece_id=7ODKEDJBK3WMG4LHBPVJE2PGMURUMQBHGMA7J2CVNXP6DK5DIL6A, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.362940474 (498h9m2.234023529s) [noise=true , address=145.79.227.44:4225 , piece_id=IUI7KC75SGIV7JY2G6KS6I3PAEJ7GOHL6ZJH5F6XQSLSE3YQLMRA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.365628162 (498h9m2.236711207s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:03.828213372 (498h9m3.699296417s) [upload=437210]: Next returned nil segment Dec 26 09:38:03.828232628 (498h9m3.699315673s) [upload=437210]: Waiting for error group managing segments... Dec 26 09:38:04.229734875 (498h9m4.10081792s) [noise=true , address=145.79.227.34:4933 , piece_id=7ODKEDJBK3WMG4LHBPVJE2PGMURUMQBHGMA7J2CVNXP6DK5DIL6A, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.251273058 (498h9m4.122356103s) [noise=true , address=145.79.227.7:4322 , piece_id=EKN2EMS5DQCO6QZNIPUSTJIOHB6S5ZXD4NCJCUW2UGRYSGZRUEYQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.252890037 (498h9m4.123973082s) [noise=true , address=145.79.227.40:4081 , piece_id=FQKTTQ4RF3CL4Z7ASHOZFINUSEKUSC3WELUAJHUK5NWQPZZ5NUQA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.359834698 (498h9m4.230917733s) [noise=true , address=64.49.13.140:4508 , piece_id=64FDQH7EMUMDSWRGXSCBD63EGZACEJKZJLXL4IYKSP7RPZHLALNA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.460384842 (498h9m4.331467887s) [noise=true , address=45.91.116.8:4059 , piece_id=VGGWQT6LBVG3I2PGQE2CGNSUETQ6PGPFYWYCGULPF2YYZB4YGUNA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.499297058 (498h9m4.370380103s) [noise=true , address=77.90.31.21:4886 , piece_id=HSVV4VVNNYY6KZR2XPAHYDVPJO6PPBMMRS4JVEOWMZAIVDDNWJGQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.542084071 (498h9m4.413167116s) [noise=true , address=77.90.31.11:4566 , piece_id=L4EZMVP56QYGCJPGWK3FQ4NO5HO7Y4KCRFWKEG2TWATBFEFIGUIA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.582574843 (498h9m4.453657878s) [noise=true , address=45.91.116.10:4100 , piece_id=SIU7MB27XG7QQWM2ZDUADCNCBKJO32TCGJPRUETYKUDZORZK32UA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.844901267 (498h9m4.715984312s) [noise=true , address=77.90.31.16:4756 , piece_id=BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.093586545 (498h9m4.96466959s) [noise=true , address=145.79.227.30:4787 , piece_id=UA4ZRVGGRT7U6MHLC3LTVPA25K6E3XLLLQ2C6766Z24GJHY4WATA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.158083007 (498h9m5.029166052s) [noise=true , address=145.79.227.44:4225 , piece_id=IUI7KC75SGIV7JY2G6KS6I3PAEJ7GOHL6ZJH5F6XQSLSE3YQLMRA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.158420730 (498h9m5.029503776s) [noise=true , address=45.91.116.9:4288 , piece_id=MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.382024590 (498h9m5.253107635s) [noise=true , address=77.90.31.18:4813 , piece_id=LMQL4MOLI2UM2AGA774XJY3FFU2C4OC66TVEBNZKRWQPGZNJ2GYA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.470838497 (498h9m5.341921542s) [noise=true , address=45.91.116.5:4330 , piece_id=YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.470865969 (498h9m5.341949004s) [seg_pos={0 0}, upload=437210]: Segment reached optimal threshold of 14 pieces. Dec 26 09:38:05.471121566 (498h9m5.342204621s) [noise=true , address=77.90.31.15:4725 , piece_id=6HXOF7P376BXRFNGCPIPHQO4ZEREMHUPM373AKMBBVZW27L7AQQA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nstPr2GMJo6pnBmBuDME5QYw3mXgnqLoaTKkcKUhjSbAY8mQk): protocol: expected piece hash; context canceled; EOF Dec 26 09:38:05.471130843 (498h9m5.342213888s) [noise=true , address=77.90.31.15:4725 , piece_id=6HXOF7P376BXRFNGCPIPHQO4ZEREMHUPM373AKMBBVZW27L7AQQA, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471200115 (498h9m5.34228315s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection Dec 26 09:38:05.471216476 (498h9m5.342299511s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471359830 (498h9m5.342442875s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection Dec 26 09:38:05.471368907 (498h9m5.342451942s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471685119 (498h9m5.342768164s) [noise=true , address=77.90.31.13:4649 , piece_id=CSRCWI2DPEM5WZXNTRQI6I72R7YJAB4K2PBPWTBMV7GDUICUFBDQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: read tcp 188.166.225.170:51276->77.90.31.13:4649: use of closed network connection; context canceled Dec 26 09:38:05.471694005 (498h9m5.342777041s) [noise=true , address=77.90.31.13:4649 , piece_id=CSRCWI2DPEM5WZXNTRQI6I72R7YJAB4K2PBPWTBMV7GDUICUFBDQ, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471712161 (498h9m5.342795206s) [seg_pos={0 0}, upload=437210]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 09:38:05.471800789 (498h9m5.342883834s) [upload=437210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:38:05.471805818 (498h9m5.342888853s) [upload=437210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:38:05.775837956 (498h9m5.646921001s) [upload=437210]: Done uploading segments... Dec 26 09:38:05.775863094 (498h9m5.646946129s) [upload=437210]: Upload finished. err: <nil> Duration: 50.576448759s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X....XXX..X.......X.....XX.XX.XXX.....X..X.....X.X...X..X] duration: 30.003583099s trace: https://jaeger.dev.storj.io/trace/51a64afca96aa72a FAIL (2025-12-26 09:41:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: TVE2FV6PAKUCOBC3F7RRPHUHKNB5JP2L7IS66YKEXMCQGEOKN5LQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: Z553SJUFUHRZRJQWRCQTF54RESPTIQ64HM7XOI4RX3FBP2ZFGH6A) context deadline exceeded ======LOGS====== Dec 26 09:40:36.418965447 (498h11m36.290048482s) [upload=437252]: Starting upload Dec 26 09:40:36.419036562 (498h11m36.290119608s) [upload=437252]: Done starting upload Dec 26 09:40:36.419266961 (498h11m36.290350006s) [upload=437252]: Uploading segments... Dec 26 09:40:36.419316375 (498h11m36.29039942s) [upload=437252]: Got next segment. Inline: false Dec 26 09:40:36.419354498 (498h11m36.290437533s) [upload=437252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:40:36.419360108 (498h11m36.290443153s) [upload=437252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:40:36.663695855 (498h11m36.5347789s) [seg_pos={0 0}, upload=437252]: Begin upload segment... Dec 26 09:40:36.663757803 (498h11m36.534840838s) [seg_pos={0 0}, upload=437252]: Done begin upload segment. Dec 26 09:40:36.663800264 (498h11m36.534883309s) [noise=true , address=145.79.227.40:4050 , piece_id=L54B72OTRXZBZ3V27LYKVJ63RZXFS45MPIATDHTOP44YIUAGKSZA, seg_pos={0 0}, upload=437252]: Uploading piece... Dec 26 09:40:36.663895375 (498h11m36.53497842s) [noise=true , address=145.79.227.4:4185 , piece_id=TGQQIY3RH3YYKNQXF2LOXFFLCO52IA5WK34RG6YMFDGVTSHGXXCA, seg_pos={0 0}, upload=437252]: Uploading piece... Dec 26 09:40:36.664011616 (498h11m36.535094651s) [noise=true , address=45.91.116.5:4298 , piece_id=DHJJRG4BIHGRDY5PPWWTHYN3REWUMH5D6BGDOZ3DYUFUMSINNRYQ, seg_pos={0 0}, upload=437252]: Uploading piece... Dec 26 09:40:36.664427839 (498h11m36.535510874s) [noise=true , address=145.79.227.49:4337 , piece_id=ZYBJLCZDDSF46CTXGKVLUTT7WDIDEQGX4XREOH32HUJQI5I5PZJA, seg_pos={0 0}, upload=437252]: Uploading piece... Dec 26 09:40:36.664540443 (498h11m36.535623478s) [noise=true , address=145.79.227.33:4871 , piece_id=WGK2VCQ7CY2RWEAT4QUB5SROOJOYLVW2PWUSSZJTZIUWPG5YHAKQ, seg_pos={0 0}, upload=437252]: Uploading piece... Dec 26 09:40:36.664833722 (498h11m36.535916757s) [noise=true , address=77.90.31.16:4736 , piece_id=METOC4RCHH6D762YKLSN4KZTGEVMVRMJROU2V3XWSUB4YP4YY6QA, seg_pos={0 0}, upload=437252]: Uploading piece... Dec 26 09:40:36.665106981 (498h11m36.536190016s) [noise=true , address=145.79.227.32:4835 , piece_id=52UGKCKLDVJ7Q7UYTM4N6OO5COOG7IELPDUCJS4K67R4OHT3F57A, seg_pos={0 0}, upload=437252]: Uploading piece... Dec 26 09:40:36.665390953 (498h11m36.536473998s) [noise=true , address=145.79.227.6:4259 , piece_id=FXP3VKPISIYPERCBTAXEDVKBQA6TPSDJUZWZHJRPW2KDCULDN7MQ, seg_pos={0 0}, upload=437252]: Uploading piece... Dec 26 09:40:36.665670485 (498h11m36.53675352s) [noise=true , address=45.91.116.13:4154 , piece_id=TVE2FV6PAKUCOBC3F7RRPHUHKNB5JP2L7IS66YKEXMCQGEOKN5LQ, seg_pos={0 0}, upload=437252]: Uploading piece... Dec 26 09:40:36.665981858 (498h11m36.537064893s) [noise=true , address=77.90.31.21:4880 , piece_id=3FFUG2AOIOHU6IO2OG4DQ66XKUYTFXKZBMWFBYG5PXKORA6RFIOA, seg_pos={0 0}, upload=437252]: Uploading piece... Dec 26 09:40:36.666305213 (498h11m36.537388248s) [noise=true , address=145.79.227.7:4295 , piece_id=TXC2RIVGKCT2NYTR4B2JNHHMBRXLN3MCFSFKDWWB4XGDBKWTSNOQ, seg_pos={0 0}, upload=437252]: Uploading piece... Dec 26 09:40:36.666579666 (498h11m36.537662701s) [noise=true , address=77.90.31.15:4697 , piece_id=PDTXR5GC27KA26FV2FRG4GWZBSX2U7DYAUWCZLML5DUO3OHI5HIQ, seg_pos={0 0}, upload=437252]: Uploading piece... Dec 26 09:40:36.670222462 (498h11m36.541305507s) [noise=true , address=77.90.31.10:4517 , piece_id=7NZZWA5QVRIS2UW6VLJRUUI5TZA7WIIIWGIWHDIZRMC4WKSU2MPA, seg_pos={0 0}, upload=437252]: Uploading piece... Dec 26 09:40:36.670761128 (498h11m36.541844173s) [noise=true , address=145.79.227.36:4517 , piece_id=BTGI7QBU3PXULLXPQBVSCGNUS45WTDRZENTLFKEVBA3B7QPLMWCA, seg_pos={0 0}, upload=437252]: Uploading piece... Dec 26 09:40:36.671260719 (498h11m36.542343764s) [noise=true , address=77.90.31.11:4555 , piece_id=H65ZFNCE7FXBRBPHUIN25JTDTHE5XQMKKX7DKQHHSQQGMWK34GVQ, seg_pos={0 0}, upload=437252]: Uploading piece... Dec 26 09:40:36.671749510 (498h11m36.542832545s) [noise=true , address=45.91.116.8:4046 , piece_id=Z553SJUFUHRZRJQWRCQTF54RESPTIQ64HM7XOI4RX3FBP2ZFGH6A, seg_pos={0 0}, upload=437252]: Uploading piece... Dec 26 09:40:37.768619276 (498h11m37.639702311s) [noise=true , address=77.90.31.15:4697 , piece_id=PDTXR5GC27KA26FV2FRG4GWZBSX2U7DYAUWCZLML5DUO3OHI5HIQ, seg_pos={0 0}, upload=437252]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 26 09:40:38.279396983 (498h11m38.150480018s) [upload=437252]: Next returned nil segment Dec 26 09:40:38.279410318 (498h11m38.150493353s) [upload=437252]: Waiting for error group managing segments... Dec 26 09:40:38.664909153 (498h11m38.535992188s) [noise=true , address=145.79.227.33:4871 , piece_id=WGK2VCQ7CY2RWEAT4QUB5SROOJOYLVW2PWUSSZJTZIUWPG5YHAKQ, seg_pos={0 0}, upload=437252]: Done uploading piece. err: <nil> Dec 26 09:40:38.669922930 (498h11m38.541005965s) [noise=true , address=145.79.227.6:4259 , piece_id=FXP3VKPISIYPERCBTAXEDVKBQA6TPSDJUZWZHJRPW2KDCULDN7MQ, seg_pos={0 0}, upload=437252]: Done uploading piece. err: <nil> Dec 26 09:40:38.679704015 (498h11m38.55078705s) [noise=true , address=145.79.227.4:4185 , piece_id=TGQQIY3RH3YYKNQXF2LOXFFLCO52IA5WK34RG6YMFDGVTSHGXXCA, seg_pos={0 0}, upload=437252]: Done uploading piece. err: <nil> Dec 26 09:40:38.680046616 (498h11m38.551129651s) [noise=true , address=145.79.227.36:4517 , piece_id=BTGI7QBU3PXULLXPQBVSCGNUS45WTDRZENTLFKEVBA3B7QPLMWCA, seg_pos={0 0}, upload=437252]: Done uploading piece. err: <nil> Dec 26 09:40:38.697330368 (498h11m38.568413403s) [noise=true , address=145.79.227.40:4050 , piece_id=L54B72OTRXZBZ3V27LYKVJ63RZXFS45MPIATDHTOP44YIUAGKSZA, seg_pos={0 0}, upload=437252]: Done uploading piece. err: <nil> Dec 26 09:40:38.703568186 (498h11m38.574651221s) [noise=true , address=145.79.227.7:4295 , piece_id=TXC2RIVGKCT2NYTR4B2JNHHMBRXLN3MCFSFKDWWB4XGDBKWTSNOQ, seg_pos={0 0}, upload=437252]: Done uploading piece. err: <nil> Dec 26 09:40:38.825536434 (498h11m38.696619469s) [noise=true , address=145.79.227.32:4835 , piece_id=52UGKCKLDVJ7Q7UYTM4N6OO5COOG7IELPDUCJS4K67R4OHT3F57A, seg_pos={0 0}, upload=437252]: Done uploading piece. err: <nil> Dec 26 09:40:38.911067550 (498h11m38.782150595s) [noise=true , address=45.91.116.8:4046 , piece_id=Z553SJUFUHRZRJQWRCQTF54RESPTIQ64HM7XOI4RX3FBP2ZFGH6A, seg_pos={0 0}, upload=437252]: Done uploading piece. err: <nil> Dec 26 09:40:38.931986253 (498h11m38.803069288s) [noise=true , address=45.91.116.13:4154 , piece_id=TVE2FV6PAKUCOBC3F7RRPHUHKNB5JP2L7IS66YKEXMCQGEOKN5LQ, seg_pos={0 0}, upload=437252]: Done uploading piece. err: <nil> Dec 26 09:40:38.992131769 (498h11m38.863214814s) [noise=true , address=77.90.31.10:4517 , piece_id=7NZZWA5QVRIS2UW6VLJRUUI5TZA7WIIIWGIWHDIZRMC4WKSU2MPA, seg_pos={0 0}, upload=437252]: Done uploading piece. err: <nil> Dec 26 09:40:38.997226431 (498h11m38.868309486s) [noise=true , address=145.79.227.49:4337 , piece_id=ZYBJLCZDDSF46CTXGKVLUTT7WDIDEQGX4XREOH32HUJQI5I5PZJA, seg_pos={0 0}, upload=437252]: Done uploading piece. err: <nil> Dec 26 09:40:38.997842383 (498h11m38.868925428s) [noise=true , address=45.91.116.5:4298 , piece_id=DHJJRG4BIHGRDY5PPWWTHYN3REWUMH5D6BGDOZ3DYUFUMSINNRYQ, seg_pos={0 0}, upload=437252]: Done uploading piece. err: <nil> Dec 26 09:40:38.998551644 (498h11m38.869634689s) [noise=true , address=77.90.31.11:4555 , piece_id=H65ZFNCE7FXBRBPHUIN25JTDTHE5XQMKKX7DKQHHSQQGMWK34GVQ, seg_pos={0 0}, upload=437252]: Done uploading piece. err: <nil> Dec 26 09:40:38.998564559 (498h11m38.869647594s) [seg_pos={0 0}, upload=437252]: Segment reached optimal threshold of 13 pieces. Dec 26 09:40:38.998813172 (498h11m38.869896237s) [noise=true , address=77.90.31.21:4880 , piece_id=3FFUG2AOIOHU6IO2OG4DQ66XKUYTFXKZBMWFBYG5PXKORA6RFIOA, seg_pos={0 0}, upload=437252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 26 09:40:38.998823292 (498h11m38.869906457s) [noise=true , address=77.90.31.21:4880 , piece_id=3FFUG2AOIOHU6IO2OG4DQ66XKUYTFXKZBMWFBYG5PXKORA6RFIOA, seg_pos={0 0}, upload=437252]: Long tail cancellation due to optimal threshold Dec 26 09:40:38.998873116 (498h11m38.869956161s) [noise=true , address=77.90.31.16:4736 , piece_id=METOC4RCHH6D762YKLSN4KZTGEVMVRMJROU2V3XWSUB4YP4YY6QA, seg_pos={0 0}, upload=437252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 09:40:38.998916990 (498h11m38.870000025s) [noise=true , address=77.90.31.16:4736 , piece_id=METOC4RCHH6D762YKLSN4KZTGEVMVRMJROU2V3XWSUB4YP4YY6QA, seg_pos={0 0}, upload=437252]: Long tail cancellation due to optimal threshold Dec 26 09:40:38.998949662 (498h11m38.870032697s) [seg_pos={0 0}, upload=437252]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:40:38.999033973 (498h11m38.870117018s) [upload=437252]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:40:38.999040545 (498h11m38.87012358s) [upload=437252]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:40:39.298117518 (498h11m39.169200563s) [upload=437252]: Done uploading segments... Dec 26 09:40:39.298201518 (498h11m39.169284554s) [upload=437252]: Upload finished. err: <nil> Duration: 30.003606474s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 53.272324753s trace: https://jaeger.dev.storj.io/trace/667977214d97ec8c FAIL (2025-12-26 09:40:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, Piece ID: YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA) context deadline exceeded; piecestore: (Node ID: 1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, Piece ID: MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A) context deadline exceeded ======LOGS====== true , address=77.90.31.11:4566 , piece_id=L4EZMVP56QYGCJPGWK3FQ4NO5HO7Y4KCRFWKEG2TWATBFEFIGUIA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.357693223 (498h9m2.228776258s) [noise=true , address=77.90.31.21:4886 , piece_id=HSVV4VVNNYY6KZR2XPAHYDVPJO6PPBMMRS4JVEOWMZAIVDDNWJGQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.358418583 (498h9m2.229501628s) [noise=true , address=64.49.13.140:4508 , piece_id=64FDQH7EMUMDSWRGXSCBD63EGZACEJKZJLXL4IYKSP7RPZHLALNA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.358935137 (498h9m2.230018182s) [noise=true , address=77.90.31.16:4756 , piece_id=BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359491216 (498h9m2.230574261s) [noise=true , address=45.91.116.9:4288 , piece_id=MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359953967 (498h9m2.231037013s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359968154 (498h9m2.231051199s) [noise=true , address=77.90.31.18:4813 , piece_id=LMQL4MOLI2UM2AGA774XJY3FFU2C4OC66TVEBNZKRWQPGZNJ2GYA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.360429653 (498h9m2.231512698s) [noise=true , address=145.79.227.40:4081 , piece_id=FQKTTQ4RF3CL4Z7ASHOZFINUSEKUSC3WELUAJHUK5NWQPZZ5NUQA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.360518873 (498h9m2.231601918s) [noise=true , address=145.79.227.7:4322 , piece_id=EKN2EMS5DQCO6QZNIPUSTJIOHB6S5ZXD4NCJCUW2UGRYSGZRUEYQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361114697 (498h9m2.232197742s) [noise=true , address=45.91.116.8:4059 , piece_id=VGGWQT6LBVG3I2PGQE2CGNSUETQ6PGPFYWYCGULPF2YYZB4YGUNA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361136940 (498h9m2.232219985s) [noise=true , address=45.91.116.5:4330 , piece_id=YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361631942 (498h9m2.232714988s) [noise=true , address=45.91.116.10:4100 , piece_id=SIU7MB27XG7QQWM2ZDUADCNCBKJO32TCGJPRUETYKUDZORZK32UA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361832193 (498h9m2.232915238s) [noise=true , address=145.79.227.34:4933 , piece_id=7ODKEDJBK3WMG4LHBPVJE2PGMURUMQBHGMA7J2CVNXP6DK5DIL6A, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.362940474 (498h9m2.234023529s) [noise=true , address=145.79.227.44:4225 , piece_id=IUI7KC75SGIV7JY2G6KS6I3PAEJ7GOHL6ZJH5F6XQSLSE3YQLMRA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.365628162 (498h9m2.236711207s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:03.828213372 (498h9m3.699296417s) [upload=437210]: Next returned nil segment Dec 26 09:38:03.828232628 (498h9m3.699315673s) [upload=437210]: Waiting for error group managing segments... Dec 26 09:38:04.229734875 (498h9m4.10081792s) [noise=true , address=145.79.227.34:4933 , piece_id=7ODKEDJBK3WMG4LHBPVJE2PGMURUMQBHGMA7J2CVNXP6DK5DIL6A, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.251273058 (498h9m4.122356103s) [noise=true , address=145.79.227.7:4322 , piece_id=EKN2EMS5DQCO6QZNIPUSTJIOHB6S5ZXD4NCJCUW2UGRYSGZRUEYQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.252890037 (498h9m4.123973082s) [noise=true , address=145.79.227.40:4081 , piece_id=FQKTTQ4RF3CL4Z7ASHOZFINUSEKUSC3WELUAJHUK5NWQPZZ5NUQA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.359834698 (498h9m4.230917733s) [noise=true , address=64.49.13.140:4508 , piece_id=64FDQH7EMUMDSWRGXSCBD63EGZACEJKZJLXL4IYKSP7RPZHLALNA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.460384842 (498h9m4.331467887s) [noise=true , address=45.91.116.8:4059 , piece_id=VGGWQT6LBVG3I2PGQE2CGNSUETQ6PGPFYWYCGULPF2YYZB4YGUNA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.499297058 (498h9m4.370380103s) [noise=true , address=77.90.31.21:4886 , piece_id=HSVV4VVNNYY6KZR2XPAHYDVPJO6PPBMMRS4JVEOWMZAIVDDNWJGQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.542084071 (498h9m4.413167116s) [noise=true , address=77.90.31.11:4566 , piece_id=L4EZMVP56QYGCJPGWK3FQ4NO5HO7Y4KCRFWKEG2TWATBFEFIGUIA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.582574843 (498h9m4.453657878s) [noise=true , address=45.91.116.10:4100 , piece_id=SIU7MB27XG7QQWM2ZDUADCNCBKJO32TCGJPRUETYKUDZORZK32UA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.844901267 (498h9m4.715984312s) [noise=true , address=77.90.31.16:4756 , piece_id=BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.093586545 (498h9m4.96466959s) [noise=true , address=145.79.227.30:4787 , piece_id=UA4ZRVGGRT7U6MHLC3LTVPA25K6E3XLLLQ2C6766Z24GJHY4WATA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.158083007 (498h9m5.029166052s) [noise=true , address=145.79.227.44:4225 , piece_id=IUI7KC75SGIV7JY2G6KS6I3PAEJ7GOHL6ZJH5F6XQSLSE3YQLMRA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.158420730 (498h9m5.029503776s) [noise=true , address=45.91.116.9:4288 , piece_id=MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.382024590 (498h9m5.253107635s) [noise=true , address=77.90.31.18:4813 , piece_id=LMQL4MOLI2UM2AGA774XJY3FFU2C4OC66TVEBNZKRWQPGZNJ2GYA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.470838497 (498h9m5.341921542s) [noise=true , address=45.91.116.5:4330 , piece_id=YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.470865969 (498h9m5.341949004s) [seg_pos={0 0}, upload=437210]: Segment reached optimal threshold of 14 pieces. Dec 26 09:38:05.471121566 (498h9m5.342204621s) [noise=true , address=77.90.31.15:4725 , piece_id=6HXOF7P376BXRFNGCPIPHQO4ZEREMHUPM373AKMBBVZW27L7AQQA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nstPr2GMJo6pnBmBuDME5QYw3mXgnqLoaTKkcKUhjSbAY8mQk): protocol: expected piece hash; context canceled; EOF Dec 26 09:38:05.471130843 (498h9m5.342213888s) [noise=true , address=77.90.31.15:4725 , piece_id=6HXOF7P376BXRFNGCPIPHQO4ZEREMHUPM373AKMBBVZW27L7AQQA, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471200115 (498h9m5.34228315s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection Dec 26 09:38:05.471216476 (498h9m5.342299511s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471359830 (498h9m5.342442875s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection Dec 26 09:38:05.471368907 (498h9m5.342451942s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471685119 (498h9m5.342768164s) [noise=true , address=77.90.31.13:4649 , piece_id=CSRCWI2DPEM5WZXNTRQI6I72R7YJAB4K2PBPWTBMV7GDUICUFBDQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: read tcp 188.166.225.170:51276->77.90.31.13:4649: use of closed network connection; context canceled Dec 26 09:38:05.471694005 (498h9m5.342777041s) [noise=true , address=77.90.31.13:4649 , piece_id=CSRCWI2DPEM5WZXNTRQI6I72R7YJAB4K2PBPWTBMV7GDUICUFBDQ, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471712161 (498h9m5.342795206s) [seg_pos={0 0}, upload=437210]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 09:38:05.471800789 (498h9m5.342883834s) [upload=437210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:38:05.471805818 (498h9m5.342888853s) [upload=437210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:38:05.775837956 (498h9m5.646921001s) [upload=437210]: Done uploading segments... Dec 26 09:38:05.775863094 (498h9m5.646946129s) [upload=437210]: Upload finished. err: <nil> Duration: 53.272345453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 48.630333186s trace: https://jaeger.dev.storj.io/trace/667977214d97ec8c FAIL (2025-12-26 09:39:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, Piece ID: YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA) context deadline exceeded; piecestore: (Node ID: 1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, Piece ID: MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A) context deadline exceeded ======LOGS====== true , address=77.90.31.11:4566 , piece_id=L4EZMVP56QYGCJPGWK3FQ4NO5HO7Y4KCRFWKEG2TWATBFEFIGUIA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.357693223 (498h9m2.228776258s) [noise=true , address=77.90.31.21:4886 , piece_id=HSVV4VVNNYY6KZR2XPAHYDVPJO6PPBMMRS4JVEOWMZAIVDDNWJGQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.358418583 (498h9m2.229501628s) [noise=true , address=64.49.13.140:4508 , piece_id=64FDQH7EMUMDSWRGXSCBD63EGZACEJKZJLXL4IYKSP7RPZHLALNA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.358935137 (498h9m2.230018182s) [noise=true , address=77.90.31.16:4756 , piece_id=BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359491216 (498h9m2.230574261s) [noise=true , address=45.91.116.9:4288 , piece_id=MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359953967 (498h9m2.231037013s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359968154 (498h9m2.231051199s) [noise=true , address=77.90.31.18:4813 , piece_id=LMQL4MOLI2UM2AGA774XJY3FFU2C4OC66TVEBNZKRWQPGZNJ2GYA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.360429653 (498h9m2.231512698s) [noise=true , address=145.79.227.40:4081 , piece_id=FQKTTQ4RF3CL4Z7ASHOZFINUSEKUSC3WELUAJHUK5NWQPZZ5NUQA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.360518873 (498h9m2.231601918s) [noise=true , address=145.79.227.7:4322 , piece_id=EKN2EMS5DQCO6QZNIPUSTJIOHB6S5ZXD4NCJCUW2UGRYSGZRUEYQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361114697 (498h9m2.232197742s) [noise=true , address=45.91.116.8:4059 , piece_id=VGGWQT6LBVG3I2PGQE2CGNSUETQ6PGPFYWYCGULPF2YYZB4YGUNA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361136940 (498h9m2.232219985s) [noise=true , address=45.91.116.5:4330 , piece_id=YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361631942 (498h9m2.232714988s) [noise=true , address=45.91.116.10:4100 , piece_id=SIU7MB27XG7QQWM2ZDUADCNCBKJO32TCGJPRUETYKUDZORZK32UA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361832193 (498h9m2.232915238s) [noise=true , address=145.79.227.34:4933 , piece_id=7ODKEDJBK3WMG4LHBPVJE2PGMURUMQBHGMA7J2CVNXP6DK5DIL6A, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.362940474 (498h9m2.234023529s) [noise=true , address=145.79.227.44:4225 , piece_id=IUI7KC75SGIV7JY2G6KS6I3PAEJ7GOHL6ZJH5F6XQSLSE3YQLMRA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.365628162 (498h9m2.236711207s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:03.828213372 (498h9m3.699296417s) [upload=437210]: Next returned nil segment Dec 26 09:38:03.828232628 (498h9m3.699315673s) [upload=437210]: Waiting for error group managing segments... Dec 26 09:38:04.229734875 (498h9m4.10081792s) [noise=true , address=145.79.227.34:4933 , piece_id=7ODKEDJBK3WMG4LHBPVJE2PGMURUMQBHGMA7J2CVNXP6DK5DIL6A, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.251273058 (498h9m4.122356103s) [noise=true , address=145.79.227.7:4322 , piece_id=EKN2EMS5DQCO6QZNIPUSTJIOHB6S5ZXD4NCJCUW2UGRYSGZRUEYQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.252890037 (498h9m4.123973082s) [noise=true , address=145.79.227.40:4081 , piece_id=FQKTTQ4RF3CL4Z7ASHOZFINUSEKUSC3WELUAJHUK5NWQPZZ5NUQA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.359834698 (498h9m4.230917733s) [noise=true , address=64.49.13.140:4508 , piece_id=64FDQH7EMUMDSWRGXSCBD63EGZACEJKZJLXL4IYKSP7RPZHLALNA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.460384842 (498h9m4.331467887s) [noise=true , address=45.91.116.8:4059 , piece_id=VGGWQT6LBVG3I2PGQE2CGNSUETQ6PGPFYWYCGULPF2YYZB4YGUNA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.499297058 (498h9m4.370380103s) [noise=true , address=77.90.31.21:4886 , piece_id=HSVV4VVNNYY6KZR2XPAHYDVPJO6PPBMMRS4JVEOWMZAIVDDNWJGQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.542084071 (498h9m4.413167116s) [noise=true , address=77.90.31.11:4566 , piece_id=L4EZMVP56QYGCJPGWK3FQ4NO5HO7Y4KCRFWKEG2TWATBFEFIGUIA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.582574843 (498h9m4.453657878s) [noise=true , address=45.91.116.10:4100 , piece_id=SIU7MB27XG7QQWM2ZDUADCNCBKJO32TCGJPRUETYKUDZORZK32UA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.844901267 (498h9m4.715984312s) [noise=true , address=77.90.31.16:4756 , piece_id=BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.093586545 (498h9m4.96466959s) [noise=true , address=145.79.227.30:4787 , piece_id=UA4ZRVGGRT7U6MHLC3LTVPA25K6E3XLLLQ2C6766Z24GJHY4WATA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.158083007 (498h9m5.029166052s) [noise=true , address=145.79.227.44:4225 , piece_id=IUI7KC75SGIV7JY2G6KS6I3PAEJ7GOHL6ZJH5F6XQSLSE3YQLMRA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.158420730 (498h9m5.029503776s) [noise=true , address=45.91.116.9:4288 , piece_id=MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.382024590 (498h9m5.253107635s) [noise=true , address=77.90.31.18:4813 , piece_id=LMQL4MOLI2UM2AGA774XJY3FFU2C4OC66TVEBNZKRWQPGZNJ2GYA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.470838497 (498h9m5.341921542s) [noise=true , address=45.91.116.5:4330 , piece_id=YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.470865969 (498h9m5.341949004s) [seg_pos={0 0}, upload=437210]: Segment reached optimal threshold of 14 pieces. Dec 26 09:38:05.471121566 (498h9m5.342204621s) [noise=true , address=77.90.31.15:4725 , piece_id=6HXOF7P376BXRFNGCPIPHQO4ZEREMHUPM373AKMBBVZW27L7AQQA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nstPr2GMJo6pnBmBuDME5QYw3mXgnqLoaTKkcKUhjSbAY8mQk): protocol: expected piece hash; context canceled; EOF Dec 26 09:38:05.471130843 (498h9m5.342213888s) [noise=true , address=77.90.31.15:4725 , piece_id=6HXOF7P376BXRFNGCPIPHQO4ZEREMHUPM373AKMBBVZW27L7AQQA, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471200115 (498h9m5.34228315s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection Dec 26 09:38:05.471216476 (498h9m5.342299511s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471359830 (498h9m5.342442875s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection Dec 26 09:38:05.471368907 (498h9m5.342451942s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471685119 (498h9m5.342768164s) [noise=true , address=77.90.31.13:4649 , piece_id=CSRCWI2DPEM5WZXNTRQI6I72R7YJAB4K2PBPWTBMV7GDUICUFBDQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: read tcp 188.166.225.170:51276->77.90.31.13:4649: use of closed network connection; context canceled Dec 26 09:38:05.471694005 (498h9m5.342777041s) [noise=true , address=77.90.31.13:4649 , piece_id=CSRCWI2DPEM5WZXNTRQI6I72R7YJAB4K2PBPWTBMV7GDUICUFBDQ, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471712161 (498h9m5.342795206s) [seg_pos={0 0}, upload=437210]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 09:38:05.471800789 (498h9m5.342883834s) [upload=437210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:38:05.471805818 (498h9m5.342888853s) [upload=437210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:38:05.775837956 (498h9m5.646921001s) [upload=437210]: Done uploading segments... Dec 26 09:38:05.775863094 (498h9m5.646946129s) [upload=437210]: Upload finished. err: <nil> Duration: 48.630350177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X....XXX..X.......X.....XX.XX.XXX.....X..X.....X.X...X..] duration: 5.668957381s trace: https://jaeger.dev.storj.io/trace/a25dcdf0ea88d4d FAIL (2025-12-26 09:39:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: J5KMT6H4N4MGQCSCV7PEDI3N2OSU6BXERZIZRIHERR67WLQMVE5Q) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 4LLAFAFROURUNTQ6NOURXGKAGG2UNMA4EKBUGIHGRLI467ICEEOA) context deadline exceeded ======LOGS====== Dec 26 09:38:01.694828406 (498h9m1.565911442s) [upload=437209]: Starting upload Dec 26 09:38:01.694896986 (498h9m1.565980031s) [upload=437209]: Done starting upload Dec 26 09:38:01.694995233 (498h9m1.566078278s) [upload=437209]: Uploading segments... Dec 26 09:38:01.695205153 (498h9m1.566288218s) [upload=437209]: Got next segment. Inline: false Dec 26 09:38:01.695264196 (498h9m1.566347241s) [upload=437209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:38:01.695271740 (498h9m1.566354795s) [upload=437209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:38:01.939654676 (498h9m1.810737741s) [seg_pos={0 0}, upload=437209]: Begin upload segment... Dec 26 09:38:01.939711845 (498h9m1.8107949s) [seg_pos={0 0}, upload=437209]: Done begin upload segment. Dec 26 09:38:01.939744507 (498h9m1.810827562s) [noise=true , address=45.91.116.9:4261 , piece_id=PQ4ERCV3L4URKC63CBB6AIAYNZKKNJK6PT2F2YJJ4PTJXDCJRT2A, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.939790304 (498h9m1.810873369s) [noise=true , address=45.91.116.13:4155 , piece_id=J5KMT6H4N4MGQCSCV7PEDI3N2OSU6BXERZIZRIHERR67WLQMVE5Q, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.941119335 (498h9m1.81220238s) [noise=true , address=45.91.116.8:4045 , piece_id=L5PVLGO7FCJZRXEFBFMFH2JRH76NAONBNCU5SWRKKIZJ43TSXCFA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.941755116 (498h9m1.812838161s) [noise=true , address=145.79.227.37:4565 , piece_id=XOGTDQI7MIYFZTWWSDGXN3IM6HQCHVESYUVFDZYAWXYILNWHQY6A, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.942443847 (498h9m1.813526892s) [noise=true , address=145.79.227.32:4835 , piece_id=B435RK4RDRM6H2XX7MOGZMSZG2CHHB2O7NUU5EDZSCYLAN6BMFRA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.943010967 (498h9m1.814094013s) [noise=true , address=77.90.31.11:4556 , piece_id=XYZWGHFGIILDBQ4FA5YGPSF2LNTSRV2B32BN7SX2XRFMTDF64WMQ, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.943509797 (498h9m1.814592842s) [noise=true , address=45.91.116.7:4010 , piece_id=WHYTAPPXOYZAIIUBIKYRQKFQADAEVMVQKJZQZY6ISJASXFET3WGQ, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.944060295 (498h9m1.81514333s) [noise=true , address=45.91.116.14:4191 , piece_id=EXFJFHZBNF4B72THVW3QM3TYU7DFHHEXUHMK4KU5KSRD366HH7XA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.946092765 (498h9m1.81717581s) [noise=true , address=45.91.116.5:4298 , piece_id=ATTT4GJTZ6MOUXXIA32OGJX62B5LOBVH3Y7YKOT7S4C6FZMOXD3Q, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.946263110 (498h9m1.817346165s) [noise=true , address=145.79.227.51:4411 , piece_id=6GSMOCR2HUJYBDT2CPTGJUL75ROSQCQXZLISDJZQJ4MWQHYKTF6Q, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.948517162 (498h9m1.819600217s) [noise=true , address=145.79.227.36:4519 , piece_id=IAT7JRED666D7FSQDWDMFSLMFODK7QIL656KTL2IKZ3A7TE7B6RA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.948529976 (498h9m1.819613031s) [noise=true , address=145.79.227.19:4078 , piece_id=WM7A5UZUCRFWQL5HHUEMZIDBY5753X23B4WIWLQFKXTNXVEN7KBQ, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.950573768 (498h9m1.821656823s) [noise=true , address=77.90.31.17:4773 , piece_id=4LLAFAFROURUNTQ6NOURXGKAGG2UNMA4EKBUGIHGRLI467ICEEOA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.950875763 (498h9m1.821958809s) [noise=true , address=145.79.227.22:4511 , piece_id=QRSWUFIGMNSXXBYN5D2CIY2XNZ5CISPQZ33WHL2ARYIWTO7CQ37A, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.952625806 (498h9m1.823708851s) [noise=true , address=77.90.31.14:4662 , piece_id=75XNOYCCRGZJKZRLOFPQUAOU4MEIOWLKV4CNPX2F6JYRPGL2CQVA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.952872926 (498h9m1.823955971s) [noise=true , address=145.79.227.35:4483 , piece_id=QOZRZCYK54QZSO5WZQUIKUUAYLJNIGB735D3EYEUJKCHVGRKIGDA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:03.675687787 (498h9m3.546770832s) [upload=437209]: Next returned nil segment Dec 26 09:38:03.675727062 (498h9m3.546810097s) [upload=437209]: Waiting for error group managing segments... Dec 26 09:38:03.956897827 (498h9m3.827980872s) [noise=true , address=145.79.227.22:4511 , piece_id=QRSWUFIGMNSXXBYN5D2CIY2XNZ5CISPQZ33WHL2ARYIWTO7CQ37A, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.957956203 (498h9m3.829039268s) [noise=true , address=145.79.227.37:4565 , piece_id=XOGTDQI7MIYFZTWWSDGXN3IM6HQCHVESYUVFDZYAWXYILNWHQY6A, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.966305061 (498h9m3.837388096s) [noise=true , address=145.79.227.36:4519 , piece_id=IAT7JRED666D7FSQDWDMFSLMFODK7QIL656KTL2IKZ3A7TE7B6RA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.969597440 (498h9m3.840680495s) [noise=true , address=145.79.227.51:4411 , piece_id=6GSMOCR2HUJYBDT2CPTGJUL75ROSQCQXZLISDJZQJ4MWQHYKTF6Q, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.979096498 (498h9m3.850179543s) [noise=true , address=145.79.227.32:4835 , piece_id=B435RK4RDRM6H2XX7MOGZMSZG2CHHB2O7NUU5EDZSCYLAN6BMFRA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.980012192 (498h9m3.851095227s) [noise=true , address=145.79.227.19:4078 , piece_id=WM7A5UZUCRFWQL5HHUEMZIDBY5753X23B4WIWLQFKXTNXVEN7KBQ, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.985148201 (498h9m3.856231246s) [noise=true , address=145.79.227.35:4483 , piece_id=QOZRZCYK54QZSO5WZQUIKUUAYLJNIGB735D3EYEUJKCHVGRKIGDA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.231554842 (498h9m4.102637887s) [noise=true , address=77.90.31.14:4662 , piece_id=75XNOYCCRGZJKZRLOFPQUAOU4MEIOWLKV4CNPX2F6JYRPGL2CQVA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.281546584 (498h9m4.152629629s) [noise=true , address=45.91.116.13:4155 , piece_id=J5KMT6H4N4MGQCSCV7PEDI3N2OSU6BXERZIZRIHERR67WLQMVE5Q, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.290361159 (498h9m4.161444224s) [noise=true , address=77.90.31.11:4556 , piece_id=XYZWGHFGIILDBQ4FA5YGPSF2LNTSRV2B32BN7SX2XRFMTDF64WMQ, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.496684012 (498h9m4.367767057s) [noise=true , address=45.91.116.7:4010 , piece_id=WHYTAPPXOYZAIIUBIKYRQKFQADAEVMVQKJZQZY6ISJASXFET3WGQ, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.829931742 (498h9m4.701014787s) [noise=true , address=77.90.31.17:4773 , piece_id=4LLAFAFROURUNTQ6NOURXGKAGG2UNMA4EKBUGIHGRLI467ICEEOA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.909658986 (498h9m4.780742041s) [noise=true , address=45.91.116.9:4261 , piece_id=PQ4ERCV3L4URKC63CBB6AIAYNZKKNJK6PT2F2YJJ4PTJXDCJRT2A, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.909681368 (498h9m4.780764413s) [seg_pos={0 0}, upload=437209]: Segment reached optimal threshold of 13 pieces. Dec 26 09:38:04.909977603 (498h9m4.781060648s) [noise=true , address=45.91.116.5:4298 , piece_id=ATTT4GJTZ6MOUXXIA32OGJX62B5LOBVH3Y7YKOT7S4C6FZMOXD3Q, seg_pos={0 0}, upload=437209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 09:38:04.909988694 (498h9m4.781071739s) [noise=true , address=45.91.116.5:4298 , piece_id=ATTT4GJTZ6MOUXXIA32OGJX62B5LOBVH3Y7YKOT7S4C6FZMOXD3Q, seg_pos={0 0}, upload=437209]: Long tail cancellation due to optimal threshold Dec 26 09:38:04.910071201 (498h9m4.781154246s) [noise=true , address=45.91.116.14:4191 , piece_id=EXFJFHZBNF4B72THVW3QM3TYU7DFHHEXUHMK4KU5KSRD366HH7XA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 09:38:04.910104394 (498h9m4.781187429s) [noise=true , address=45.91.116.14:4191 , piece_id=EXFJFHZBNF4B72THVW3QM3TYU7DFHHEXUHMK4KU5KSRD366HH7XA, seg_pos={0 0}, upload=437209]: Long tail cancellation due to optimal threshold Dec 26 09:38:04.910319514 (498h9m4.781402559s) [noise=true , address=45.91.116.8:4045 , piece_id=L5PVLGO7FCJZRXEFBFMFH2JRH76NAONBNCU5SWRKKIZJ43TSXCFA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:51224->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:51224->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51224->45.91.116.8:4045: use of closed network connection Dec 26 09:38:04.910337257 (498h9m4.781420302s) [noise=true , address=45.91.116.8:4045 , piece_id=L5PVLGO7FCJZRXEFBFMFH2JRH76NAONBNCU5SWRKKIZJ43TSXCFA, seg_pos={0 0}, upload=437209]: Long tail cancellation due to optimal threshold Dec 26 09:38:04.910358378 (498h9m4.781441423s) [seg_pos={0 0}, upload=437209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:38:04.910434743 (498h9m4.781517788s) [upload=437209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:38:04.910439732 (498h9m4.781522778s) [upload=437209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:38:05.214956264 (498h9m5.086039329s) [upload=437209]: Done uploading segments... Dec 26 09:38:05.214983085 (498h9m5.08606613s) [upload=437209]: Upload finished. err: <nil> Duration: 5.66897836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.....X....XXX..X.......X.....XX.XX.XXX.....X..X.....X.X...X.] duration: 5.715232568s trace: https://jaeger.dev.storj.io/trace/a25dcdf0ea88d4d FAIL (2025-12-26 09:39:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: J5KMT6H4N4MGQCSCV7PEDI3N2OSU6BXERZIZRIHERR67WLQMVE5Q) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 4LLAFAFROURUNTQ6NOURXGKAGG2UNMA4EKBUGIHGRLI467ICEEOA) context deadline exceeded ======LOGS====== Dec 26 09:38:01.694828406 (498h9m1.565911442s) [upload=437209]: Starting upload Dec 26 09:38:01.694896986 (498h9m1.565980031s) [upload=437209]: Done starting upload Dec 26 09:38:01.694995233 (498h9m1.566078278s) [upload=437209]: Uploading segments... Dec 26 09:38:01.695205153 (498h9m1.566288218s) [upload=437209]: Got next segment. Inline: false Dec 26 09:38:01.695264196 (498h9m1.566347241s) [upload=437209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:38:01.695271740 (498h9m1.566354795s) [upload=437209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:38:01.939654676 (498h9m1.810737741s) [seg_pos={0 0}, upload=437209]: Begin upload segment... Dec 26 09:38:01.939711845 (498h9m1.8107949s) [seg_pos={0 0}, upload=437209]: Done begin upload segment. Dec 26 09:38:01.939744507 (498h9m1.810827562s) [noise=true , address=45.91.116.9:4261 , piece_id=PQ4ERCV3L4URKC63CBB6AIAYNZKKNJK6PT2F2YJJ4PTJXDCJRT2A, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.939790304 (498h9m1.810873369s) [noise=true , address=45.91.116.13:4155 , piece_id=J5KMT6H4N4MGQCSCV7PEDI3N2OSU6BXERZIZRIHERR67WLQMVE5Q, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.941119335 (498h9m1.81220238s) [noise=true , address=45.91.116.8:4045 , piece_id=L5PVLGO7FCJZRXEFBFMFH2JRH76NAONBNCU5SWRKKIZJ43TSXCFA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.941755116 (498h9m1.812838161s) [noise=true , address=145.79.227.37:4565 , piece_id=XOGTDQI7MIYFZTWWSDGXN3IM6HQCHVESYUVFDZYAWXYILNWHQY6A, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.942443847 (498h9m1.813526892s) [noise=true , address=145.79.227.32:4835 , piece_id=B435RK4RDRM6H2XX7MOGZMSZG2CHHB2O7NUU5EDZSCYLAN6BMFRA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.943010967 (498h9m1.814094013s) [noise=true , address=77.90.31.11:4556 , piece_id=XYZWGHFGIILDBQ4FA5YGPSF2LNTSRV2B32BN7SX2XRFMTDF64WMQ, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.943509797 (498h9m1.814592842s) [noise=true , address=45.91.116.7:4010 , piece_id=WHYTAPPXOYZAIIUBIKYRQKFQADAEVMVQKJZQZY6ISJASXFET3WGQ, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.944060295 (498h9m1.81514333s) [noise=true , address=45.91.116.14:4191 , piece_id=EXFJFHZBNF4B72THVW3QM3TYU7DFHHEXUHMK4KU5KSRD366HH7XA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.946092765 (498h9m1.81717581s) [noise=true , address=45.91.116.5:4298 , piece_id=ATTT4GJTZ6MOUXXIA32OGJX62B5LOBVH3Y7YKOT7S4C6FZMOXD3Q, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.946263110 (498h9m1.817346165s) [noise=true , address=145.79.227.51:4411 , piece_id=6GSMOCR2HUJYBDT2CPTGJUL75ROSQCQXZLISDJZQJ4MWQHYKTF6Q, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.948517162 (498h9m1.819600217s) [noise=true , address=145.79.227.36:4519 , piece_id=IAT7JRED666D7FSQDWDMFSLMFODK7QIL656KTL2IKZ3A7TE7B6RA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.948529976 (498h9m1.819613031s) [noise=true , address=145.79.227.19:4078 , piece_id=WM7A5UZUCRFWQL5HHUEMZIDBY5753X23B4WIWLQFKXTNXVEN7KBQ, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.950573768 (498h9m1.821656823s) [noise=true , address=77.90.31.17:4773 , piece_id=4LLAFAFROURUNTQ6NOURXGKAGG2UNMA4EKBUGIHGRLI467ICEEOA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.950875763 (498h9m1.821958809s) [noise=true , address=145.79.227.22:4511 , piece_id=QRSWUFIGMNSXXBYN5D2CIY2XNZ5CISPQZ33WHL2ARYIWTO7CQ37A, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.952625806 (498h9m1.823708851s) [noise=true , address=77.90.31.14:4662 , piece_id=75XNOYCCRGZJKZRLOFPQUAOU4MEIOWLKV4CNPX2F6JYRPGL2CQVA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.952872926 (498h9m1.823955971s) [noise=true , address=145.79.227.35:4483 , piece_id=QOZRZCYK54QZSO5WZQUIKUUAYLJNIGB735D3EYEUJKCHVGRKIGDA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:03.675687787 (498h9m3.546770832s) [upload=437209]: Next returned nil segment Dec 26 09:38:03.675727062 (498h9m3.546810097s) [upload=437209]: Waiting for error group managing segments... Dec 26 09:38:03.956897827 (498h9m3.827980872s) [noise=true , address=145.79.227.22:4511 , piece_id=QRSWUFIGMNSXXBYN5D2CIY2XNZ5CISPQZ33WHL2ARYIWTO7CQ37A, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.957956203 (498h9m3.829039268s) [noise=true , address=145.79.227.37:4565 , piece_id=XOGTDQI7MIYFZTWWSDGXN3IM6HQCHVESYUVFDZYAWXYILNWHQY6A, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.966305061 (498h9m3.837388096s) [noise=true , address=145.79.227.36:4519 , piece_id=IAT7JRED666D7FSQDWDMFSLMFODK7QIL656KTL2IKZ3A7TE7B6RA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.969597440 (498h9m3.840680495s) [noise=true , address=145.79.227.51:4411 , piece_id=6GSMOCR2HUJYBDT2CPTGJUL75ROSQCQXZLISDJZQJ4MWQHYKTF6Q, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.979096498 (498h9m3.850179543s) [noise=true , address=145.79.227.32:4835 , piece_id=B435RK4RDRM6H2XX7MOGZMSZG2CHHB2O7NUU5EDZSCYLAN6BMFRA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.980012192 (498h9m3.851095227s) [noise=true , address=145.79.227.19:4078 , piece_id=WM7A5UZUCRFWQL5HHUEMZIDBY5753X23B4WIWLQFKXTNXVEN7KBQ, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.985148201 (498h9m3.856231246s) [noise=true , address=145.79.227.35:4483 , piece_id=QOZRZCYK54QZSO5WZQUIKUUAYLJNIGB735D3EYEUJKCHVGRKIGDA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.231554842 (498h9m4.102637887s) [noise=true , address=77.90.31.14:4662 , piece_id=75XNOYCCRGZJKZRLOFPQUAOU4MEIOWLKV4CNPX2F6JYRPGL2CQVA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.281546584 (498h9m4.152629629s) [noise=true , address=45.91.116.13:4155 , piece_id=J5KMT6H4N4MGQCSCV7PEDI3N2OSU6BXERZIZRIHERR67WLQMVE5Q, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.290361159 (498h9m4.161444224s) [noise=true , address=77.90.31.11:4556 , piece_id=XYZWGHFGIILDBQ4FA5YGPSF2LNTSRV2B32BN7SX2XRFMTDF64WMQ, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.496684012 (498h9m4.367767057s) [noise=true , address=45.91.116.7:4010 , piece_id=WHYTAPPXOYZAIIUBIKYRQKFQADAEVMVQKJZQZY6ISJASXFET3WGQ, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.829931742 (498h9m4.701014787s) [noise=true , address=77.90.31.17:4773 , piece_id=4LLAFAFROURUNTQ6NOURXGKAGG2UNMA4EKBUGIHGRLI467ICEEOA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.909658986 (498h9m4.780742041s) [noise=true , address=45.91.116.9:4261 , piece_id=PQ4ERCV3L4URKC63CBB6AIAYNZKKNJK6PT2F2YJJ4PTJXDCJRT2A, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.909681368 (498h9m4.780764413s) [seg_pos={0 0}, upload=437209]: Segment reached optimal threshold of 13 pieces. Dec 26 09:38:04.909977603 (498h9m4.781060648s) [noise=true , address=45.91.116.5:4298 , piece_id=ATTT4GJTZ6MOUXXIA32OGJX62B5LOBVH3Y7YKOT7S4C6FZMOXD3Q, seg_pos={0 0}, upload=437209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 09:38:04.909988694 (498h9m4.781071739s) [noise=true , address=45.91.116.5:4298 , piece_id=ATTT4GJTZ6MOUXXIA32OGJX62B5LOBVH3Y7YKOT7S4C6FZMOXD3Q, seg_pos={0 0}, upload=437209]: Long tail cancellation due to optimal threshold Dec 26 09:38:04.910071201 (498h9m4.781154246s) [noise=true , address=45.91.116.14:4191 , piece_id=EXFJFHZBNF4B72THVW3QM3TYU7DFHHEXUHMK4KU5KSRD366HH7XA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 09:38:04.910104394 (498h9m4.781187429s) [noise=true , address=45.91.116.14:4191 , piece_id=EXFJFHZBNF4B72THVW3QM3TYU7DFHHEXUHMK4KU5KSRD366HH7XA, seg_pos={0 0}, upload=437209]: Long tail cancellation due to optimal threshold Dec 26 09:38:04.910319514 (498h9m4.781402559s) [noise=true , address=45.91.116.8:4045 , piece_id=L5PVLGO7FCJZRXEFBFMFH2JRH76NAONBNCU5SWRKKIZJ43TSXCFA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:51224->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:51224->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51224->45.91.116.8:4045: use of closed network connection Dec 26 09:38:04.910337257 (498h9m4.781420302s) [noise=true , address=45.91.116.8:4045 , piece_id=L5PVLGO7FCJZRXEFBFMFH2JRH76NAONBNCU5SWRKKIZJ43TSXCFA, seg_pos={0 0}, upload=437209]: Long tail cancellation due to optimal threshold Dec 26 09:38:04.910358378 (498h9m4.781441423s) [seg_pos={0 0}, upload=437209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:38:04.910434743 (498h9m4.781517788s) [upload=437209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:38:04.910439732 (498h9m4.781522778s) [upload=437209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:38:05.214956264 (498h9m5.086039329s) [upload=437209]: Done uploading segments... Dec 26 09:38:05.214983085 (498h9m5.08606613s) [upload=437209]: Upload finished. err: <nil> Duration: 5.715262083s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.006562596s trace: https://jaeger.dev.storj.io/trace/667977214d97ec8c FAIL (2025-12-26 09:39:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, Piece ID: YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA) context deadline exceeded; piecestore: (Node ID: 1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, Piece ID: MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A) context deadline exceeded ======LOGS====== true , address=77.90.31.11:4566 , piece_id=L4EZMVP56QYGCJPGWK3FQ4NO5HO7Y4KCRFWKEG2TWATBFEFIGUIA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.357693223 (498h9m2.228776258s) [noise=true , address=77.90.31.21:4886 , piece_id=HSVV4VVNNYY6KZR2XPAHYDVPJO6PPBMMRS4JVEOWMZAIVDDNWJGQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.358418583 (498h9m2.229501628s) [noise=true , address=64.49.13.140:4508 , piece_id=64FDQH7EMUMDSWRGXSCBD63EGZACEJKZJLXL4IYKSP7RPZHLALNA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.358935137 (498h9m2.230018182s) [noise=true , address=77.90.31.16:4756 , piece_id=BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359491216 (498h9m2.230574261s) [noise=true , address=45.91.116.9:4288 , piece_id=MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359953967 (498h9m2.231037013s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.359968154 (498h9m2.231051199s) [noise=true , address=77.90.31.18:4813 , piece_id=LMQL4MOLI2UM2AGA774XJY3FFU2C4OC66TVEBNZKRWQPGZNJ2GYA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.360429653 (498h9m2.231512698s) [noise=true , address=145.79.227.40:4081 , piece_id=FQKTTQ4RF3CL4Z7ASHOZFINUSEKUSC3WELUAJHUK5NWQPZZ5NUQA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.360518873 (498h9m2.231601918s) [noise=true , address=145.79.227.7:4322 , piece_id=EKN2EMS5DQCO6QZNIPUSTJIOHB6S5ZXD4NCJCUW2UGRYSGZRUEYQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361114697 (498h9m2.232197742s) [noise=true , address=45.91.116.8:4059 , piece_id=VGGWQT6LBVG3I2PGQE2CGNSUETQ6PGPFYWYCGULPF2YYZB4YGUNA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361136940 (498h9m2.232219985s) [noise=true , address=45.91.116.5:4330 , piece_id=YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361631942 (498h9m2.232714988s) [noise=true , address=45.91.116.10:4100 , piece_id=SIU7MB27XG7QQWM2ZDUADCNCBKJO32TCGJPRUETYKUDZORZK32UA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.361832193 (498h9m2.232915238s) [noise=true , address=145.79.227.34:4933 , piece_id=7ODKEDJBK3WMG4LHBPVJE2PGMURUMQBHGMA7J2CVNXP6DK5DIL6A, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.362940474 (498h9m2.234023529s) [noise=true , address=145.79.227.44:4225 , piece_id=IUI7KC75SGIV7JY2G6KS6I3PAEJ7GOHL6ZJH5F6XQSLSE3YQLMRA, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:02.365628162 (498h9m2.236711207s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Uploading piece... Dec 26 09:38:03.828213372 (498h9m3.699296417s) [upload=437210]: Next returned nil segment Dec 26 09:38:03.828232628 (498h9m3.699315673s) [upload=437210]: Waiting for error group managing segments... Dec 26 09:38:04.229734875 (498h9m4.10081792s) [noise=true , address=145.79.227.34:4933 , piece_id=7ODKEDJBK3WMG4LHBPVJE2PGMURUMQBHGMA7J2CVNXP6DK5DIL6A, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.251273058 (498h9m4.122356103s) [noise=true , address=145.79.227.7:4322 , piece_id=EKN2EMS5DQCO6QZNIPUSTJIOHB6S5ZXD4NCJCUW2UGRYSGZRUEYQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.252890037 (498h9m4.123973082s) [noise=true , address=145.79.227.40:4081 , piece_id=FQKTTQ4RF3CL4Z7ASHOZFINUSEKUSC3WELUAJHUK5NWQPZZ5NUQA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.359834698 (498h9m4.230917733s) [noise=true , address=64.49.13.140:4508 , piece_id=64FDQH7EMUMDSWRGXSCBD63EGZACEJKZJLXL4IYKSP7RPZHLALNA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.460384842 (498h9m4.331467887s) [noise=true , address=45.91.116.8:4059 , piece_id=VGGWQT6LBVG3I2PGQE2CGNSUETQ6PGPFYWYCGULPF2YYZB4YGUNA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.499297058 (498h9m4.370380103s) [noise=true , address=77.90.31.21:4886 , piece_id=HSVV4VVNNYY6KZR2XPAHYDVPJO6PPBMMRS4JVEOWMZAIVDDNWJGQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.542084071 (498h9m4.413167116s) [noise=true , address=77.90.31.11:4566 , piece_id=L4EZMVP56QYGCJPGWK3FQ4NO5HO7Y4KCRFWKEG2TWATBFEFIGUIA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.582574843 (498h9m4.453657878s) [noise=true , address=45.91.116.10:4100 , piece_id=SIU7MB27XG7QQWM2ZDUADCNCBKJO32TCGJPRUETYKUDZORZK32UA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:04.844901267 (498h9m4.715984312s) [noise=true , address=77.90.31.16:4756 , piece_id=BVIVTAAAU4SZEYOAVFCCORP5RV4CUUQSJB4MVDQMZF4QR3FVAXVA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.093586545 (498h9m4.96466959s) [noise=true , address=145.79.227.30:4787 , piece_id=UA4ZRVGGRT7U6MHLC3LTVPA25K6E3XLLLQ2C6766Z24GJHY4WATA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.158083007 (498h9m5.029166052s) [noise=true , address=145.79.227.44:4225 , piece_id=IUI7KC75SGIV7JY2G6KS6I3PAEJ7GOHL6ZJH5F6XQSLSE3YQLMRA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.158420730 (498h9m5.029503776s) [noise=true , address=45.91.116.9:4288 , piece_id=MSYUDKJQ4OQIJWNFUDCJL6EZHU7MT2J2UJQS56W6GHTYTOBDNR3A, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.382024590 (498h9m5.253107635s) [noise=true , address=77.90.31.18:4813 , piece_id=LMQL4MOLI2UM2AGA774XJY3FFU2C4OC66TVEBNZKRWQPGZNJ2GYA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.470838497 (498h9m5.341921542s) [noise=true , address=45.91.116.5:4330 , piece_id=YEG36RXZ4XR5TD3LTTZ4Q3QWKFCAN3QY2FRNHZALHIQJDQVHS63Q, seg_pos={0 0}, upload=437210]: Done uploading piece. err: <nil> Dec 26 09:38:05.470865969 (498h9m5.341949004s) [seg_pos={0 0}, upload=437210]: Segment reached optimal threshold of 14 pieces. Dec 26 09:38:05.471121566 (498h9m5.342204621s) [noise=true , address=77.90.31.15:4725 , piece_id=6HXOF7P376BXRFNGCPIPHQO4ZEREMHUPM373AKMBBVZW27L7AQQA, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nstPr2GMJo6pnBmBuDME5QYw3mXgnqLoaTKkcKUhjSbAY8mQk): protocol: expected piece hash; context canceled; EOF Dec 26 09:38:05.471130843 (498h9m5.342213888s) [noise=true , address=77.90.31.15:4725 , piece_id=6HXOF7P376BXRFNGCPIPHQO4ZEREMHUPM373AKMBBVZW27L7AQQA, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471200115 (498h9m5.34228315s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52596->64.49.13.169:4715: use of closed network connection Dec 26 09:38:05.471216476 (498h9m5.342299511s) [noise=true , address=64.49.13.169:4715 , piece_id=YUJOESED4UKSEHE3X3A6B5KZKG6QMCR33JDCEJBHEP37FGUOD4VQ, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471359830 (498h9m5.342442875s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52744->64.49.13.169:4707: use of closed network connection Dec 26 09:38:05.471368907 (498h9m5.342451942s) [noise=true , address=64.49.13.169:4707 , piece_id=XUPGD7TMJ7JP6C2QCFN7KDU2FSPENEIVS4ZYS3E5N6DE3AQ2WN4Q, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471685119 (498h9m5.342768164s) [noise=true , address=77.90.31.13:4649 , piece_id=CSRCWI2DPEM5WZXNTRQI6I72R7YJAB4K2PBPWTBMV7GDUICUFBDQ, seg_pos={0 0}, upload=437210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: read tcp 188.166.225.170:51276->77.90.31.13:4649: use of closed network connection; context canceled Dec 26 09:38:05.471694005 (498h9m5.342777041s) [noise=true , address=77.90.31.13:4649 , piece_id=CSRCWI2DPEM5WZXNTRQI6I72R7YJAB4K2PBPWTBMV7GDUICUFBDQ, seg_pos={0 0}, upload=437210]: Long tail cancellation due to optimal threshold Dec 26 09:38:05.471712161 (498h9m5.342795206s) [seg_pos={0 0}, upload=437210]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 09:38:05.471800789 (498h9m5.342883834s) [upload=437210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:38:05.471805818 (498h9m5.342888853s) [upload=437210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:38:05.775837956 (498h9m5.646921001s) [upload=437210]: Done uploading segments... Dec 26 09:38:05.775863094 (498h9m5.646946129s) [upload=437210]: Upload finished. err: <nil> Duration: 1m0.006579188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [......X....XXX..X.......X.....XX.XX.XXX.....X..X.....X.X...X] duration: 30.003288814s trace: https://jaeger.dev.storj.io/trace/a25dcdf0ea88d4d FAIL (2025-12-26 09:38:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: J5KMT6H4N4MGQCSCV7PEDI3N2OSU6BXERZIZRIHERR67WLQMVE5Q) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 4LLAFAFROURUNTQ6NOURXGKAGG2UNMA4EKBUGIHGRLI467ICEEOA) context deadline exceeded ======LOGS====== Dec 26 09:38:01.694828406 (498h9m1.565911442s) [upload=437209]: Starting upload Dec 26 09:38:01.694896986 (498h9m1.565980031s) [upload=437209]: Done starting upload Dec 26 09:38:01.694995233 (498h9m1.566078278s) [upload=437209]: Uploading segments... Dec 26 09:38:01.695205153 (498h9m1.566288218s) [upload=437209]: Got next segment. Inline: false Dec 26 09:38:01.695264196 (498h9m1.566347241s) [upload=437209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:38:01.695271740 (498h9m1.566354795s) [upload=437209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:38:01.939654676 (498h9m1.810737741s) [seg_pos={0 0}, upload=437209]: Begin upload segment... Dec 26 09:38:01.939711845 (498h9m1.8107949s) [seg_pos={0 0}, upload=437209]: Done begin upload segment. Dec 26 09:38:01.939744507 (498h9m1.810827562s) [noise=true , address=45.91.116.9:4261 , piece_id=PQ4ERCV3L4URKC63CBB6AIAYNZKKNJK6PT2F2YJJ4PTJXDCJRT2A, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.939790304 (498h9m1.810873369s) [noise=true , address=45.91.116.13:4155 , piece_id=J5KMT6H4N4MGQCSCV7PEDI3N2OSU6BXERZIZRIHERR67WLQMVE5Q, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.941119335 (498h9m1.81220238s) [noise=true , address=45.91.116.8:4045 , piece_id=L5PVLGO7FCJZRXEFBFMFH2JRH76NAONBNCU5SWRKKIZJ43TSXCFA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.941755116 (498h9m1.812838161s) [noise=true , address=145.79.227.37:4565 , piece_id=XOGTDQI7MIYFZTWWSDGXN3IM6HQCHVESYUVFDZYAWXYILNWHQY6A, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.942443847 (498h9m1.813526892s) [noise=true , address=145.79.227.32:4835 , piece_id=B435RK4RDRM6H2XX7MOGZMSZG2CHHB2O7NUU5EDZSCYLAN6BMFRA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.943010967 (498h9m1.814094013s) [noise=true , address=77.90.31.11:4556 , piece_id=XYZWGHFGIILDBQ4FA5YGPSF2LNTSRV2B32BN7SX2XRFMTDF64WMQ, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.943509797 (498h9m1.814592842s) [noise=true , address=45.91.116.7:4010 , piece_id=WHYTAPPXOYZAIIUBIKYRQKFQADAEVMVQKJZQZY6ISJASXFET3WGQ, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.944060295 (498h9m1.81514333s) [noise=true , address=45.91.116.14:4191 , piece_id=EXFJFHZBNF4B72THVW3QM3TYU7DFHHEXUHMK4KU5KSRD366HH7XA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.946092765 (498h9m1.81717581s) [noise=true , address=45.91.116.5:4298 , piece_id=ATTT4GJTZ6MOUXXIA32OGJX62B5LOBVH3Y7YKOT7S4C6FZMOXD3Q, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.946263110 (498h9m1.817346165s) [noise=true , address=145.79.227.51:4411 , piece_id=6GSMOCR2HUJYBDT2CPTGJUL75ROSQCQXZLISDJZQJ4MWQHYKTF6Q, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.948517162 (498h9m1.819600217s) [noise=true , address=145.79.227.36:4519 , piece_id=IAT7JRED666D7FSQDWDMFSLMFODK7QIL656KTL2IKZ3A7TE7B6RA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.948529976 (498h9m1.819613031s) [noise=true , address=145.79.227.19:4078 , piece_id=WM7A5UZUCRFWQL5HHUEMZIDBY5753X23B4WIWLQFKXTNXVEN7KBQ, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.950573768 (498h9m1.821656823s) [noise=true , address=77.90.31.17:4773 , piece_id=4LLAFAFROURUNTQ6NOURXGKAGG2UNMA4EKBUGIHGRLI467ICEEOA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.950875763 (498h9m1.821958809s) [noise=true , address=145.79.227.22:4511 , piece_id=QRSWUFIGMNSXXBYN5D2CIY2XNZ5CISPQZ33WHL2ARYIWTO7CQ37A, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.952625806 (498h9m1.823708851s) [noise=true , address=77.90.31.14:4662 , piece_id=75XNOYCCRGZJKZRLOFPQUAOU4MEIOWLKV4CNPX2F6JYRPGL2CQVA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:01.952872926 (498h9m1.823955971s) [noise=true , address=145.79.227.35:4483 , piece_id=QOZRZCYK54QZSO5WZQUIKUUAYLJNIGB735D3EYEUJKCHVGRKIGDA, seg_pos={0 0}, upload=437209]: Uploading piece... Dec 26 09:38:03.675687787 (498h9m3.546770832s) [upload=437209]: Next returned nil segment Dec 26 09:38:03.675727062 (498h9m3.546810097s) [upload=437209]: Waiting for error group managing segments... Dec 26 09:38:03.956897827 (498h9m3.827980872s) [noise=true , address=145.79.227.22:4511 , piece_id=QRSWUFIGMNSXXBYN5D2CIY2XNZ5CISPQZ33WHL2ARYIWTO7CQ37A, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.957956203 (498h9m3.829039268s) [noise=true , address=145.79.227.37:4565 , piece_id=XOGTDQI7MIYFZTWWSDGXN3IM6HQCHVESYUVFDZYAWXYILNWHQY6A, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.966305061 (498h9m3.837388096s) [noise=true , address=145.79.227.36:4519 , piece_id=IAT7JRED666D7FSQDWDMFSLMFODK7QIL656KTL2IKZ3A7TE7B6RA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.969597440 (498h9m3.840680495s) [noise=true , address=145.79.227.51:4411 , piece_id=6GSMOCR2HUJYBDT2CPTGJUL75ROSQCQXZLISDJZQJ4MWQHYKTF6Q, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.979096498 (498h9m3.850179543s) [noise=true , address=145.79.227.32:4835 , piece_id=B435RK4RDRM6H2XX7MOGZMSZG2CHHB2O7NUU5EDZSCYLAN6BMFRA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.980012192 (498h9m3.851095227s) [noise=true , address=145.79.227.19:4078 , piece_id=WM7A5UZUCRFWQL5HHUEMZIDBY5753X23B4WIWLQFKXTNXVEN7KBQ, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:03.985148201 (498h9m3.856231246s) [noise=true , address=145.79.227.35:4483 , piece_id=QOZRZCYK54QZSO5WZQUIKUUAYLJNIGB735D3EYEUJKCHVGRKIGDA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.231554842 (498h9m4.102637887s) [noise=true , address=77.90.31.14:4662 , piece_id=75XNOYCCRGZJKZRLOFPQUAOU4MEIOWLKV4CNPX2F6JYRPGL2CQVA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.281546584 (498h9m4.152629629s) [noise=true , address=45.91.116.13:4155 , piece_id=J5KMT6H4N4MGQCSCV7PEDI3N2OSU6BXERZIZRIHERR67WLQMVE5Q, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.290361159 (498h9m4.161444224s) [noise=true , address=77.90.31.11:4556 , piece_id=XYZWGHFGIILDBQ4FA5YGPSF2LNTSRV2B32BN7SX2XRFMTDF64WMQ, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.496684012 (498h9m4.367767057s) [noise=true , address=45.91.116.7:4010 , piece_id=WHYTAPPXOYZAIIUBIKYRQKFQADAEVMVQKJZQZY6ISJASXFET3WGQ, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.829931742 (498h9m4.701014787s) [noise=true , address=77.90.31.17:4773 , piece_id=4LLAFAFROURUNTQ6NOURXGKAGG2UNMA4EKBUGIHGRLI467ICEEOA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.909658986 (498h9m4.780742041s) [noise=true , address=45.91.116.9:4261 , piece_id=PQ4ERCV3L4URKC63CBB6AIAYNZKKNJK6PT2F2YJJ4PTJXDCJRT2A, seg_pos={0 0}, upload=437209]: Done uploading piece. err: <nil> Dec 26 09:38:04.909681368 (498h9m4.780764413s) [seg_pos={0 0}, upload=437209]: Segment reached optimal threshold of 13 pieces. Dec 26 09:38:04.909977603 (498h9m4.781060648s) [noise=true , address=45.91.116.5:4298 , piece_id=ATTT4GJTZ6MOUXXIA32OGJX62B5LOBVH3Y7YKOT7S4C6FZMOXD3Q, seg_pos={0 0}, upload=437209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 09:38:04.909988694 (498h9m4.781071739s) [noise=true , address=45.91.116.5:4298 , piece_id=ATTT4GJTZ6MOUXXIA32OGJX62B5LOBVH3Y7YKOT7S4C6FZMOXD3Q, seg_pos={0 0}, upload=437209]: Long tail cancellation due to optimal threshold Dec 26 09:38:04.910071201 (498h9m4.781154246s) [noise=true , address=45.91.116.14:4191 , piece_id=EXFJFHZBNF4B72THVW3QM3TYU7DFHHEXUHMK4KU5KSRD366HH7XA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 09:38:04.910104394 (498h9m4.781187429s) [noise=true , address=45.91.116.14:4191 , piece_id=EXFJFHZBNF4B72THVW3QM3TYU7DFHHEXUHMK4KU5KSRD366HH7XA, seg_pos={0 0}, upload=437209]: Long tail cancellation due to optimal threshold Dec 26 09:38:04.910319514 (498h9m4.781402559s) [noise=true , address=45.91.116.8:4045 , piece_id=L5PVLGO7FCJZRXEFBFMFH2JRH76NAONBNCU5SWRKKIZJ43TSXCFA, seg_pos={0 0}, upload=437209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:51224->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:51224->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51224->45.91.116.8:4045: use of closed network connection Dec 26 09:38:04.910337257 (498h9m4.781420302s) [noise=true , address=45.91.116.8:4045 , piece_id=L5PVLGO7FCJZRXEFBFMFH2JRH76NAONBNCU5SWRKKIZJ43TSXCFA, seg_pos={0 0}, upload=437209]: Long tail cancellation due to optimal threshold Dec 26 09:38:04.910358378 (498h9m4.781441423s) [seg_pos={0 0}, upload=437209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:38:04.910434743 (498h9m4.781517788s) [upload=437209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:38:04.910439732 (498h9m4.781522778s) [upload=437209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:38:05.214956264 (498h9m5.086039329s) [upload=437209]: Done uploading segments... Dec 26 09:38:05.214983085 (498h9m5.08606613s) [upload=437209]: Upload finished. err: <nil> Duration: 30.003314734s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.......X....XXX..X.......X.....XX.XX.XXX.....X..X.....X.X...] duration: 5.742480353s trace: https://jaeger.dev.storj.io/trace/270c62dd941bda4e FAIL (2025-12-26 09:37:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== em: pb.BatchRequestItem_ObjectBegin Dec 26 09:35:16.004322534 (498h6m15.875405579s) [upload=437174]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:35:16.254418318 (498h6m16.125501363s) [seg_pos={0 0}, upload=437174]: Begin upload segment... Dec 26 09:35:16.254472481 (498h6m16.125555516s) [seg_pos={0 0}, upload=437174]: Done begin upload segment. Dec 26 09:35:16.254501186 (498h6m16.125584221s) [noise=true , address=145.79.227.38:4977 , piece_id=DVYIHZUGPPF3U32OKVH6WZ5WY2QQ4EKT72EQ4KYYFOXGMKSW6AXA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.255585831 (498h6m16.126668876s) [noise=true , address=145.79.227.42:4121 , piece_id=WKT4RMHTYT6C2REQEK2H66CLIBAHU2YSTC2VWAIVUE3HTHERJZYQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.256439997 (498h6m16.127523042s) [noise=true , address=77.90.31.21:4880 , piece_id=QGUUHNSH4BNIWDAYY7GIPGJ77ZNPKU573RTLDCCB5BLTJ3VLIHXA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.256853495 (498h6m16.12793654s) [noise=true , address=45.91.116.8:4046 , piece_id=YXCXBZBG74ZMCIUNP2JTT2E2SS233NFV44PUVS2TASJIJSJPQVEQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.257211177 (498h6m16.128294222s) [noise=true , address=145.79.227.11:4730 , piece_id=6TMUM6RD3DW5LZW6FZBZ32VTGS73US66GK7WTMG6I2NU7PGSUZ5A, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.257629984 (498h6m16.128713019s) [noise=true , address=145.79.227.49:4339 , piece_id=2TD6L3H4KJBIZHTWE5L4CGTA4USDHCB6CYH326BHN4KCNTNKXOKA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.258033802 (498h6m16.129116838s) [noise=true , address=145.79.227.5:4223 , piece_id=DYJAX6QEQ4WQIS27QTGL2BIUIRHZGDSGYAAXYHRC5P4G5JJ6SNUQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.258428103 (498h6m16.129511148s) [noise=true , address=77.90.31.16:4735 , piece_id=CYYL53C4CVIGLFVYGXEC2YCV4UDVO2ZUTKCGLT3Z7K64ORTTJNJA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.259028257 (498h6m16.130111292s) [noise=true , address=145.79.227.20:4437 , piece_id=XVTOLS4KJIAKU7BNOQGD4OHALHQSCZXOFX54NAJV66IH3PWVJCOA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.259643067 (498h6m16.130726102s) [noise=true , address=45.91.116.7:4009 , piece_id=3B6VMLZ22PDZOX67IXYRNGJSXEC2MCE25OQG5Y5WXC62SYBQTW2Q, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.260283437 (498h6m16.131366472s) [noise=true , address=145.79.227.2:4115 , piece_id=7DYZGFLSFO37F7NKMOCE2VVLQZLANAMYHOXLFVNHST6RJXRWZXTA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.260811061 (498h6m16.131894106s) [noise=true , address=145.79.227.3:4151 , piece_id=E3COLRH263KGXAC443GE2N5E5RBV7J5QHMTXFGPUQDR2UODD7JRQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.261255127 (498h6m16.132338162s) [noise=true , address=45.91.116.11:4120 , piece_id=HCFRGCA3J3A3NAUI67B337CIK2D3PLX4IEHRWKAJDPWNZYH43BJQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.261626674 (498h6m16.132709709s) [noise=true , address=145.79.227.50:4375 , piece_id=PZBMCZJEAINZHLQ3NFIFOOGQWSEO7KW2XAEK2MFWRMBVPQJOTUFA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.262067834 (498h6m16.133150869s) [noise=true , address=145.79.227.17:4970 , piece_id=FIGCXN5LWETJTXJINA6KBP64YABJOHNDZH4J3IAGCTBXFDF3MB7Q, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.262541617 (498h6m16.133624652s) [noise=true , address=145.79.227.29:4943 , piece_id=JPJOOOJJ6TPCH4WWCGE2475DCKCVO7MVGD2W2QIQ62UWN433IZWQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:17.979391138 (498h6m17.850474173s) [upload=437174]: Next returned nil segment Dec 26 09:35:17.979411085 (498h6m17.85049412s) [upload=437174]: Waiting for error group managing segments... Dec 26 09:35:18.258617823 (498h6m18.129700858s) [noise=true , address=145.79.227.49:4339 , piece_id=2TD6L3H4KJBIZHTWE5L4CGTA4USDHCB6CYH326BHN4KCNTNKXOKA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.268190440 (498h6m18.139273486s) [noise=true , address=145.79.227.38:4977 , piece_id=DVYIHZUGPPF3U32OKVH6WZ5WY2QQ4EKT72EQ4KYYFOXGMKSW6AXA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.269233687 (498h6m18.140316722s) [noise=true , address=145.79.227.20:4437 , piece_id=XVTOLS4KJIAKU7BNOQGD4OHALHQSCZXOFX54NAJV66IH3PWVJCOA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.271259695 (498h6m18.14234274s) [noise=true , address=145.79.227.17:4970 , piece_id=FIGCXN5LWETJTXJINA6KBP64YABJOHNDZH4J3IAGCTBXFDF3MB7Q, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.273165534 (498h6m18.144248569s) [noise=true , address=145.79.227.2:4115 , piece_id=7DYZGFLSFO37F7NKMOCE2VVLQZLANAMYHOXLFVNHST6RJXRWZXTA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.282900861 (498h6m18.153983906s) [noise=true , address=145.79.227.29:4943 , piece_id=JPJOOOJJ6TPCH4WWCGE2475DCKCVO7MVGD2W2QIQ62UWN433IZWQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.289175830 (498h6m18.160258865s) [noise=true , address=145.79.227.50:4375 , piece_id=PZBMCZJEAINZHLQ3NFIFOOGQWSEO7KW2XAEK2MFWRMBVPQJOTUFA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.289275219 (498h6m18.160358254s) [noise=true , address=145.79.227.5:4223 , piece_id=DYJAX6QEQ4WQIS27QTGL2BIUIRHZGDSGYAAXYHRC5P4G5JJ6SNUQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.291396248 (498h6m18.162479293s) [noise=true , address=145.79.227.11:4730 , piece_id=6TMUM6RD3DW5LZW6FZBZ32VTGS73US66GK7WTMG6I2NU7PGSUZ5A, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.295015770 (498h6m18.166098805s) [noise=true , address=145.79.227.42:4121 , piece_id=WKT4RMHTYT6C2REQEK2H66CLIBAHU2YSTC2VWAIVUE3HTHERJZYQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.302430649 (498h6m18.173513694s) [noise=true , address=145.79.227.3:4151 , piece_id=E3COLRH263KGXAC443GE2N5E5RBV7J5QHMTXFGPUQDR2UODD7JRQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.622465599 (498h6m18.493548644s) [noise=true , address=45.91.116.11:4120 , piece_id=HCFRGCA3J3A3NAUI67B337CIK2D3PLX4IEHRWKAJDPWNZYH43BJQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.796939017 (498h6m18.668022052s) [noise=true , address=45.91.116.8:4046 , piece_id=YXCXBZBG74ZMCIUNP2JTT2E2SS233NFV44PUVS2TASJIJSJPQVEQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.796958463 (498h6m18.668041498s) [seg_pos={0 0}, upload=437174]: Segment reached optimal threshold of 13 pieces. Dec 26 09:35:18.797131523 (498h6m18.668214558s) [noise=true , address=77.90.31.21:4880 , piece_id=QGUUHNSH4BNIWDAYY7GIPGJ77ZNPKU573RTLDCCB5BLTJ3VLIHXA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:38320->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:38320->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38320->77.90.31.21:4880: use of closed network connection Dec 26 09:35:18.797137575 (498h6m18.66822061s) [noise=true , address=77.90.31.21:4880 , piece_id=QGUUHNSH4BNIWDAYY7GIPGJ77ZNPKU573RTLDCCB5BLTJ3VLIHXA, seg_pos={0 0}, upload=437174]: Long tail cancellation due to optimal threshold Dec 26 09:35:18.797323549 (498h6m18.668406594s) [noise=true , address=77.90.31.16:4735 , piece_id=CYYL53C4CVIGLFVYGXEC2YCV4UDVO2ZUTKCGLT3Z7K64ORTTJNJA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:59758->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:59758->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59758->77.90.31.16:4735: use of closed network connection Dec 26 09:35:18.797345491 (498h6m18.668428556s) [noise=true , address=77.90.31.16:4735 , piece_id=CYYL53C4CVIGLFVYGXEC2YCV4UDVO2ZUTKCGLT3Z7K64ORTTJNJA, seg_pos={0 0}, upload=437174]: Long tail cancellation due to optimal threshold Dec 26 09:35:18.797379085 (498h6m18.66846218s) [noise=true , address=45.91.116.7:4009 , piece_id=3B6VMLZ22PDZOX67IXYRNGJSXEC2MCE25OQG5Y5WXC62SYBQTW2Q, seg_pos={0 0}, upload=437174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:57640->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:57640->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57640->45.91.116.7:4009: use of closed network connection Dec 26 09:35:18.797390957 (498h6m18.668474002s) [noise=true , address=45.91.116.7:4009 , piece_id=3B6VMLZ22PDZOX67IXYRNGJSXEC2MCE25OQG5Y5WXC62SYBQTW2Q, seg_pos={0 0}, upload=437174]: Long tail cancellation due to optimal threshold Dec 26 09:35:18.797447504 (498h6m18.668530539s) [seg_pos={0 0}, upload=437174]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:35:18.797552184 (498h6m18.668635219s) [upload=437174]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:35:18.797557735 (498h6m18.66864078s) [upload=437174]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:35:19.099664237 (498h6m18.970747272s) [upload=437174]: Done uploading segments... Dec 26 09:35:19.099752113 (498h6m18.970835158s) [upload=437174]: Upload finished. err: <nil> Duration: 5.742510279s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [........X....XXX..X.......X.....XX.XX.XXX.....X..X.....X.X..] duration: 6.003837954s trace: https://jaeger.dev.storj.io/trace/270c62dd941bda4e FAIL (2025-12-26 09:37:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== em: pb.BatchRequestItem_ObjectBegin Dec 26 09:35:16.004322534 (498h6m15.875405579s) [upload=437174]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:35:16.254418318 (498h6m16.125501363s) [seg_pos={0 0}, upload=437174]: Begin upload segment... Dec 26 09:35:16.254472481 (498h6m16.125555516s) [seg_pos={0 0}, upload=437174]: Done begin upload segment. Dec 26 09:35:16.254501186 (498h6m16.125584221s) [noise=true , address=145.79.227.38:4977 , piece_id=DVYIHZUGPPF3U32OKVH6WZ5WY2QQ4EKT72EQ4KYYFOXGMKSW6AXA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.255585831 (498h6m16.126668876s) [noise=true , address=145.79.227.42:4121 , piece_id=WKT4RMHTYT6C2REQEK2H66CLIBAHU2YSTC2VWAIVUE3HTHERJZYQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.256439997 (498h6m16.127523042s) [noise=true , address=77.90.31.21:4880 , piece_id=QGUUHNSH4BNIWDAYY7GIPGJ77ZNPKU573RTLDCCB5BLTJ3VLIHXA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.256853495 (498h6m16.12793654s) [noise=true , address=45.91.116.8:4046 , piece_id=YXCXBZBG74ZMCIUNP2JTT2E2SS233NFV44PUVS2TASJIJSJPQVEQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.257211177 (498h6m16.128294222s) [noise=true , address=145.79.227.11:4730 , piece_id=6TMUM6RD3DW5LZW6FZBZ32VTGS73US66GK7WTMG6I2NU7PGSUZ5A, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.257629984 (498h6m16.128713019s) [noise=true , address=145.79.227.49:4339 , piece_id=2TD6L3H4KJBIZHTWE5L4CGTA4USDHCB6CYH326BHN4KCNTNKXOKA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.258033802 (498h6m16.129116838s) [noise=true , address=145.79.227.5:4223 , piece_id=DYJAX6QEQ4WQIS27QTGL2BIUIRHZGDSGYAAXYHRC5P4G5JJ6SNUQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.258428103 (498h6m16.129511148s) [noise=true , address=77.90.31.16:4735 , piece_id=CYYL53C4CVIGLFVYGXEC2YCV4UDVO2ZUTKCGLT3Z7K64ORTTJNJA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.259028257 (498h6m16.130111292s) [noise=true , address=145.79.227.20:4437 , piece_id=XVTOLS4KJIAKU7BNOQGD4OHALHQSCZXOFX54NAJV66IH3PWVJCOA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.259643067 (498h6m16.130726102s) [noise=true , address=45.91.116.7:4009 , piece_id=3B6VMLZ22PDZOX67IXYRNGJSXEC2MCE25OQG5Y5WXC62SYBQTW2Q, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.260283437 (498h6m16.131366472s) [noise=true , address=145.79.227.2:4115 , piece_id=7DYZGFLSFO37F7NKMOCE2VVLQZLANAMYHOXLFVNHST6RJXRWZXTA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.260811061 (498h6m16.131894106s) [noise=true , address=145.79.227.3:4151 , piece_id=E3COLRH263KGXAC443GE2N5E5RBV7J5QHMTXFGPUQDR2UODD7JRQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.261255127 (498h6m16.132338162s) [noise=true , address=45.91.116.11:4120 , piece_id=HCFRGCA3J3A3NAUI67B337CIK2D3PLX4IEHRWKAJDPWNZYH43BJQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.261626674 (498h6m16.132709709s) [noise=true , address=145.79.227.50:4375 , piece_id=PZBMCZJEAINZHLQ3NFIFOOGQWSEO7KW2XAEK2MFWRMBVPQJOTUFA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.262067834 (498h6m16.133150869s) [noise=true , address=145.79.227.17:4970 , piece_id=FIGCXN5LWETJTXJINA6KBP64YABJOHNDZH4J3IAGCTBXFDF3MB7Q, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.262541617 (498h6m16.133624652s) [noise=true , address=145.79.227.29:4943 , piece_id=JPJOOOJJ6TPCH4WWCGE2475DCKCVO7MVGD2W2QIQ62UWN433IZWQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:17.979391138 (498h6m17.850474173s) [upload=437174]: Next returned nil segment Dec 26 09:35:17.979411085 (498h6m17.85049412s) [upload=437174]: Waiting for error group managing segments... Dec 26 09:35:18.258617823 (498h6m18.129700858s) [noise=true , address=145.79.227.49:4339 , piece_id=2TD6L3H4KJBIZHTWE5L4CGTA4USDHCB6CYH326BHN4KCNTNKXOKA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.268190440 (498h6m18.139273486s) [noise=true , address=145.79.227.38:4977 , piece_id=DVYIHZUGPPF3U32OKVH6WZ5WY2QQ4EKT72EQ4KYYFOXGMKSW6AXA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.269233687 (498h6m18.140316722s) [noise=true , address=145.79.227.20:4437 , piece_id=XVTOLS4KJIAKU7BNOQGD4OHALHQSCZXOFX54NAJV66IH3PWVJCOA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.271259695 (498h6m18.14234274s) [noise=true , address=145.79.227.17:4970 , piece_id=FIGCXN5LWETJTXJINA6KBP64YABJOHNDZH4J3IAGCTBXFDF3MB7Q, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.273165534 (498h6m18.144248569s) [noise=true , address=145.79.227.2:4115 , piece_id=7DYZGFLSFO37F7NKMOCE2VVLQZLANAMYHOXLFVNHST6RJXRWZXTA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.282900861 (498h6m18.153983906s) [noise=true , address=145.79.227.29:4943 , piece_id=JPJOOOJJ6TPCH4WWCGE2475DCKCVO7MVGD2W2QIQ62UWN433IZWQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.289175830 (498h6m18.160258865s) [noise=true , address=145.79.227.50:4375 , piece_id=PZBMCZJEAINZHLQ3NFIFOOGQWSEO7KW2XAEK2MFWRMBVPQJOTUFA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.289275219 (498h6m18.160358254s) [noise=true , address=145.79.227.5:4223 , piece_id=DYJAX6QEQ4WQIS27QTGL2BIUIRHZGDSGYAAXYHRC5P4G5JJ6SNUQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.291396248 (498h6m18.162479293s) [noise=true , address=145.79.227.11:4730 , piece_id=6TMUM6RD3DW5LZW6FZBZ32VTGS73US66GK7WTMG6I2NU7PGSUZ5A, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.295015770 (498h6m18.166098805s) [noise=true , address=145.79.227.42:4121 , piece_id=WKT4RMHTYT6C2REQEK2H66CLIBAHU2YSTC2VWAIVUE3HTHERJZYQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.302430649 (498h6m18.173513694s) [noise=true , address=145.79.227.3:4151 , piece_id=E3COLRH263KGXAC443GE2N5E5RBV7J5QHMTXFGPUQDR2UODD7JRQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.622465599 (498h6m18.493548644s) [noise=true , address=45.91.116.11:4120 , piece_id=HCFRGCA3J3A3NAUI67B337CIK2D3PLX4IEHRWKAJDPWNZYH43BJQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.796939017 (498h6m18.668022052s) [noise=true , address=45.91.116.8:4046 , piece_id=YXCXBZBG74ZMCIUNP2JTT2E2SS233NFV44PUVS2TASJIJSJPQVEQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.796958463 (498h6m18.668041498s) [seg_pos={0 0}, upload=437174]: Segment reached optimal threshold of 13 pieces. Dec 26 09:35:18.797131523 (498h6m18.668214558s) [noise=true , address=77.90.31.21:4880 , piece_id=QGUUHNSH4BNIWDAYY7GIPGJ77ZNPKU573RTLDCCB5BLTJ3VLIHXA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:38320->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:38320->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38320->77.90.31.21:4880: use of closed network connection Dec 26 09:35:18.797137575 (498h6m18.66822061s) [noise=true , address=77.90.31.21:4880 , piece_id=QGUUHNSH4BNIWDAYY7GIPGJ77ZNPKU573RTLDCCB5BLTJ3VLIHXA, seg_pos={0 0}, upload=437174]: Long tail cancellation due to optimal threshold Dec 26 09:35:18.797323549 (498h6m18.668406594s) [noise=true , address=77.90.31.16:4735 , piece_id=CYYL53C4CVIGLFVYGXEC2YCV4UDVO2ZUTKCGLT3Z7K64ORTTJNJA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:59758->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:59758->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59758->77.90.31.16:4735: use of closed network connection Dec 26 09:35:18.797345491 (498h6m18.668428556s) [noise=true , address=77.90.31.16:4735 , piece_id=CYYL53C4CVIGLFVYGXEC2YCV4UDVO2ZUTKCGLT3Z7K64ORTTJNJA, seg_pos={0 0}, upload=437174]: Long tail cancellation due to optimal threshold Dec 26 09:35:18.797379085 (498h6m18.66846218s) [noise=true , address=45.91.116.7:4009 , piece_id=3B6VMLZ22PDZOX67IXYRNGJSXEC2MCE25OQG5Y5WXC62SYBQTW2Q, seg_pos={0 0}, upload=437174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:57640->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:57640->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57640->45.91.116.7:4009: use of closed network connection Dec 26 09:35:18.797390957 (498h6m18.668474002s) [noise=true , address=45.91.116.7:4009 , piece_id=3B6VMLZ22PDZOX67IXYRNGJSXEC2MCE25OQG5Y5WXC62SYBQTW2Q, seg_pos={0 0}, upload=437174]: Long tail cancellation due to optimal threshold Dec 26 09:35:18.797447504 (498h6m18.668530539s) [seg_pos={0 0}, upload=437174]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:35:18.797552184 (498h6m18.668635219s) [upload=437174]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:35:18.797557735 (498h6m18.66864078s) [upload=437174]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:35:19.099664237 (498h6m18.970747272s) [upload=437174]: Done uploading segments... Dec 26 09:35:19.099752113 (498h6m18.970835158s) [upload=437174]: Upload finished. err: <nil> Duration: 6.003867951s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.........X....XXX..X.......X.....XX.XX.XXX.....X..X.....X.X.] duration: 5.725196281s trace: https://jaeger.dev.storj.io/trace/270c62dd941bda4e FAIL (2025-12-26 09:36:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== em: pb.BatchRequestItem_ObjectBegin Dec 26 09:35:16.004322534 (498h6m15.875405579s) [upload=437174]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:35:16.254418318 (498h6m16.125501363s) [seg_pos={0 0}, upload=437174]: Begin upload segment... Dec 26 09:35:16.254472481 (498h6m16.125555516s) [seg_pos={0 0}, upload=437174]: Done begin upload segment. Dec 26 09:35:16.254501186 (498h6m16.125584221s) [noise=true , address=145.79.227.38:4977 , piece_id=DVYIHZUGPPF3U32OKVH6WZ5WY2QQ4EKT72EQ4KYYFOXGMKSW6AXA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.255585831 (498h6m16.126668876s) [noise=true , address=145.79.227.42:4121 , piece_id=WKT4RMHTYT6C2REQEK2H66CLIBAHU2YSTC2VWAIVUE3HTHERJZYQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.256439997 (498h6m16.127523042s) [noise=true , address=77.90.31.21:4880 , piece_id=QGUUHNSH4BNIWDAYY7GIPGJ77ZNPKU573RTLDCCB5BLTJ3VLIHXA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.256853495 (498h6m16.12793654s) [noise=true , address=45.91.116.8:4046 , piece_id=YXCXBZBG74ZMCIUNP2JTT2E2SS233NFV44PUVS2TASJIJSJPQVEQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.257211177 (498h6m16.128294222s) [noise=true , address=145.79.227.11:4730 , piece_id=6TMUM6RD3DW5LZW6FZBZ32VTGS73US66GK7WTMG6I2NU7PGSUZ5A, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.257629984 (498h6m16.128713019s) [noise=true , address=145.79.227.49:4339 , piece_id=2TD6L3H4KJBIZHTWE5L4CGTA4USDHCB6CYH326BHN4KCNTNKXOKA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.258033802 (498h6m16.129116838s) [noise=true , address=145.79.227.5:4223 , piece_id=DYJAX6QEQ4WQIS27QTGL2BIUIRHZGDSGYAAXYHRC5P4G5JJ6SNUQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.258428103 (498h6m16.129511148s) [noise=true , address=77.90.31.16:4735 , piece_id=CYYL53C4CVIGLFVYGXEC2YCV4UDVO2ZUTKCGLT3Z7K64ORTTJNJA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.259028257 (498h6m16.130111292s) [noise=true , address=145.79.227.20:4437 , piece_id=XVTOLS4KJIAKU7BNOQGD4OHALHQSCZXOFX54NAJV66IH3PWVJCOA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.259643067 (498h6m16.130726102s) [noise=true , address=45.91.116.7:4009 , piece_id=3B6VMLZ22PDZOX67IXYRNGJSXEC2MCE25OQG5Y5WXC62SYBQTW2Q, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.260283437 (498h6m16.131366472s) [noise=true , address=145.79.227.2:4115 , piece_id=7DYZGFLSFO37F7NKMOCE2VVLQZLANAMYHOXLFVNHST6RJXRWZXTA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.260811061 (498h6m16.131894106s) [noise=true , address=145.79.227.3:4151 , piece_id=E3COLRH263KGXAC443GE2N5E5RBV7J5QHMTXFGPUQDR2UODD7JRQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.261255127 (498h6m16.132338162s) [noise=true , address=45.91.116.11:4120 , piece_id=HCFRGCA3J3A3NAUI67B337CIK2D3PLX4IEHRWKAJDPWNZYH43BJQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.261626674 (498h6m16.132709709s) [noise=true , address=145.79.227.50:4375 , piece_id=PZBMCZJEAINZHLQ3NFIFOOGQWSEO7KW2XAEK2MFWRMBVPQJOTUFA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.262067834 (498h6m16.133150869s) [noise=true , address=145.79.227.17:4970 , piece_id=FIGCXN5LWETJTXJINA6KBP64YABJOHNDZH4J3IAGCTBXFDF3MB7Q, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.262541617 (498h6m16.133624652s) [noise=true , address=145.79.227.29:4943 , piece_id=JPJOOOJJ6TPCH4WWCGE2475DCKCVO7MVGD2W2QIQ62UWN433IZWQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:17.979391138 (498h6m17.850474173s) [upload=437174]: Next returned nil segment Dec 26 09:35:17.979411085 (498h6m17.85049412s) [upload=437174]: Waiting for error group managing segments... Dec 26 09:35:18.258617823 (498h6m18.129700858s) [noise=true , address=145.79.227.49:4339 , piece_id=2TD6L3H4KJBIZHTWE5L4CGTA4USDHCB6CYH326BHN4KCNTNKXOKA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.268190440 (498h6m18.139273486s) [noise=true , address=145.79.227.38:4977 , piece_id=DVYIHZUGPPF3U32OKVH6WZ5WY2QQ4EKT72EQ4KYYFOXGMKSW6AXA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.269233687 (498h6m18.140316722s) [noise=true , address=145.79.227.20:4437 , piece_id=XVTOLS4KJIAKU7BNOQGD4OHALHQSCZXOFX54NAJV66IH3PWVJCOA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.271259695 (498h6m18.14234274s) [noise=true , address=145.79.227.17:4970 , piece_id=FIGCXN5LWETJTXJINA6KBP64YABJOHNDZH4J3IAGCTBXFDF3MB7Q, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.273165534 (498h6m18.144248569s) [noise=true , address=145.79.227.2:4115 , piece_id=7DYZGFLSFO37F7NKMOCE2VVLQZLANAMYHOXLFVNHST6RJXRWZXTA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.282900861 (498h6m18.153983906s) [noise=true , address=145.79.227.29:4943 , piece_id=JPJOOOJJ6TPCH4WWCGE2475DCKCVO7MVGD2W2QIQ62UWN433IZWQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.289175830 (498h6m18.160258865s) [noise=true , address=145.79.227.50:4375 , piece_id=PZBMCZJEAINZHLQ3NFIFOOGQWSEO7KW2XAEK2MFWRMBVPQJOTUFA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.289275219 (498h6m18.160358254s) [noise=true , address=145.79.227.5:4223 , piece_id=DYJAX6QEQ4WQIS27QTGL2BIUIRHZGDSGYAAXYHRC5P4G5JJ6SNUQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.291396248 (498h6m18.162479293s) [noise=true , address=145.79.227.11:4730 , piece_id=6TMUM6RD3DW5LZW6FZBZ32VTGS73US66GK7WTMG6I2NU7PGSUZ5A, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.295015770 (498h6m18.166098805s) [noise=true , address=145.79.227.42:4121 , piece_id=WKT4RMHTYT6C2REQEK2H66CLIBAHU2YSTC2VWAIVUE3HTHERJZYQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.302430649 (498h6m18.173513694s) [noise=true , address=145.79.227.3:4151 , piece_id=E3COLRH263KGXAC443GE2N5E5RBV7J5QHMTXFGPUQDR2UODD7JRQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.622465599 (498h6m18.493548644s) [noise=true , address=45.91.116.11:4120 , piece_id=HCFRGCA3J3A3NAUI67B337CIK2D3PLX4IEHRWKAJDPWNZYH43BJQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.796939017 (498h6m18.668022052s) [noise=true , address=45.91.116.8:4046 , piece_id=YXCXBZBG74ZMCIUNP2JTT2E2SS233NFV44PUVS2TASJIJSJPQVEQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.796958463 (498h6m18.668041498s) [seg_pos={0 0}, upload=437174]: Segment reached optimal threshold of 13 pieces. Dec 26 09:35:18.797131523 (498h6m18.668214558s) [noise=true , address=77.90.31.21:4880 , piece_id=QGUUHNSH4BNIWDAYY7GIPGJ77ZNPKU573RTLDCCB5BLTJ3VLIHXA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:38320->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:38320->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38320->77.90.31.21:4880: use of closed network connection Dec 26 09:35:18.797137575 (498h6m18.66822061s) [noise=true , address=77.90.31.21:4880 , piece_id=QGUUHNSH4BNIWDAYY7GIPGJ77ZNPKU573RTLDCCB5BLTJ3VLIHXA, seg_pos={0 0}, upload=437174]: Long tail cancellation due to optimal threshold Dec 26 09:35:18.797323549 (498h6m18.668406594s) [noise=true , address=77.90.31.16:4735 , piece_id=CYYL53C4CVIGLFVYGXEC2YCV4UDVO2ZUTKCGLT3Z7K64ORTTJNJA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:59758->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:59758->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59758->77.90.31.16:4735: use of closed network connection Dec 26 09:35:18.797345491 (498h6m18.668428556s) [noise=true , address=77.90.31.16:4735 , piece_id=CYYL53C4CVIGLFVYGXEC2YCV4UDVO2ZUTKCGLT3Z7K64ORTTJNJA, seg_pos={0 0}, upload=437174]: Long tail cancellation due to optimal threshold Dec 26 09:35:18.797379085 (498h6m18.66846218s) [noise=true , address=45.91.116.7:4009 , piece_id=3B6VMLZ22PDZOX67IXYRNGJSXEC2MCE25OQG5Y5WXC62SYBQTW2Q, seg_pos={0 0}, upload=437174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:57640->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:57640->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57640->45.91.116.7:4009: use of closed network connection Dec 26 09:35:18.797390957 (498h6m18.668474002s) [noise=true , address=45.91.116.7:4009 , piece_id=3B6VMLZ22PDZOX67IXYRNGJSXEC2MCE25OQG5Y5WXC62SYBQTW2Q, seg_pos={0 0}, upload=437174]: Long tail cancellation due to optimal threshold Dec 26 09:35:18.797447504 (498h6m18.668530539s) [seg_pos={0 0}, upload=437174]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:35:18.797552184 (498h6m18.668635219s) [upload=437174]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:35:18.797557735 (498h6m18.66864078s) [upload=437174]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:35:19.099664237 (498h6m18.970747272s) [upload=437174]: Done uploading segments... Dec 26 09:35:19.099752113 (498h6m18.970835158s) [upload=437174]: Upload finished. err: <nil> Duration: 5.725222241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..........X....XXX..X.......X.....XX.XX.XXX.....X..X.....X.X] duration: 30.003726049s trace: https://jaeger.dev.storj.io/trace/270c62dd941bda4e FAIL (2025-12-26 09:35:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== em: pb.BatchRequestItem_ObjectBegin Dec 26 09:35:16.004322534 (498h6m15.875405579s) [upload=437174]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:35:16.254418318 (498h6m16.125501363s) [seg_pos={0 0}, upload=437174]: Begin upload segment... Dec 26 09:35:16.254472481 (498h6m16.125555516s) [seg_pos={0 0}, upload=437174]: Done begin upload segment. Dec 26 09:35:16.254501186 (498h6m16.125584221s) [noise=true , address=145.79.227.38:4977 , piece_id=DVYIHZUGPPF3U32OKVH6WZ5WY2QQ4EKT72EQ4KYYFOXGMKSW6AXA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.255585831 (498h6m16.126668876s) [noise=true , address=145.79.227.42:4121 , piece_id=WKT4RMHTYT6C2REQEK2H66CLIBAHU2YSTC2VWAIVUE3HTHERJZYQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.256439997 (498h6m16.127523042s) [noise=true , address=77.90.31.21:4880 , piece_id=QGUUHNSH4BNIWDAYY7GIPGJ77ZNPKU573RTLDCCB5BLTJ3VLIHXA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.256853495 (498h6m16.12793654s) [noise=true , address=45.91.116.8:4046 , piece_id=YXCXBZBG74ZMCIUNP2JTT2E2SS233NFV44PUVS2TASJIJSJPQVEQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.257211177 (498h6m16.128294222s) [noise=true , address=145.79.227.11:4730 , piece_id=6TMUM6RD3DW5LZW6FZBZ32VTGS73US66GK7WTMG6I2NU7PGSUZ5A, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.257629984 (498h6m16.128713019s) [noise=true , address=145.79.227.49:4339 , piece_id=2TD6L3H4KJBIZHTWE5L4CGTA4USDHCB6CYH326BHN4KCNTNKXOKA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.258033802 (498h6m16.129116838s) [noise=true , address=145.79.227.5:4223 , piece_id=DYJAX6QEQ4WQIS27QTGL2BIUIRHZGDSGYAAXYHRC5P4G5JJ6SNUQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.258428103 (498h6m16.129511148s) [noise=true , address=77.90.31.16:4735 , piece_id=CYYL53C4CVIGLFVYGXEC2YCV4UDVO2ZUTKCGLT3Z7K64ORTTJNJA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.259028257 (498h6m16.130111292s) [noise=true , address=145.79.227.20:4437 , piece_id=XVTOLS4KJIAKU7BNOQGD4OHALHQSCZXOFX54NAJV66IH3PWVJCOA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.259643067 (498h6m16.130726102s) [noise=true , address=45.91.116.7:4009 , piece_id=3B6VMLZ22PDZOX67IXYRNGJSXEC2MCE25OQG5Y5WXC62SYBQTW2Q, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.260283437 (498h6m16.131366472s) [noise=true , address=145.79.227.2:4115 , piece_id=7DYZGFLSFO37F7NKMOCE2VVLQZLANAMYHOXLFVNHST6RJXRWZXTA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.260811061 (498h6m16.131894106s) [noise=true , address=145.79.227.3:4151 , piece_id=E3COLRH263KGXAC443GE2N5E5RBV7J5QHMTXFGPUQDR2UODD7JRQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.261255127 (498h6m16.132338162s) [noise=true , address=45.91.116.11:4120 , piece_id=HCFRGCA3J3A3NAUI67B337CIK2D3PLX4IEHRWKAJDPWNZYH43BJQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.261626674 (498h6m16.132709709s) [noise=true , address=145.79.227.50:4375 , piece_id=PZBMCZJEAINZHLQ3NFIFOOGQWSEO7KW2XAEK2MFWRMBVPQJOTUFA, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.262067834 (498h6m16.133150869s) [noise=true , address=145.79.227.17:4970 , piece_id=FIGCXN5LWETJTXJINA6KBP64YABJOHNDZH4J3IAGCTBXFDF3MB7Q, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:16.262541617 (498h6m16.133624652s) [noise=true , address=145.79.227.29:4943 , piece_id=JPJOOOJJ6TPCH4WWCGE2475DCKCVO7MVGD2W2QIQ62UWN433IZWQ, seg_pos={0 0}, upload=437174]: Uploading piece... Dec 26 09:35:17.979391138 (498h6m17.850474173s) [upload=437174]: Next returned nil segment Dec 26 09:35:17.979411085 (498h6m17.85049412s) [upload=437174]: Waiting for error group managing segments... Dec 26 09:35:18.258617823 (498h6m18.129700858s) [noise=true , address=145.79.227.49:4339 , piece_id=2TD6L3H4KJBIZHTWE5L4CGTA4USDHCB6CYH326BHN4KCNTNKXOKA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.268190440 (498h6m18.139273486s) [noise=true , address=145.79.227.38:4977 , piece_id=DVYIHZUGPPF3U32OKVH6WZ5WY2QQ4EKT72EQ4KYYFOXGMKSW6AXA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.269233687 (498h6m18.140316722s) [noise=true , address=145.79.227.20:4437 , piece_id=XVTOLS4KJIAKU7BNOQGD4OHALHQSCZXOFX54NAJV66IH3PWVJCOA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.271259695 (498h6m18.14234274s) [noise=true , address=145.79.227.17:4970 , piece_id=FIGCXN5LWETJTXJINA6KBP64YABJOHNDZH4J3IAGCTBXFDF3MB7Q, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.273165534 (498h6m18.144248569s) [noise=true , address=145.79.227.2:4115 , piece_id=7DYZGFLSFO37F7NKMOCE2VVLQZLANAMYHOXLFVNHST6RJXRWZXTA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.282900861 (498h6m18.153983906s) [noise=true , address=145.79.227.29:4943 , piece_id=JPJOOOJJ6TPCH4WWCGE2475DCKCVO7MVGD2W2QIQ62UWN433IZWQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.289175830 (498h6m18.160258865s) [noise=true , address=145.79.227.50:4375 , piece_id=PZBMCZJEAINZHLQ3NFIFOOGQWSEO7KW2XAEK2MFWRMBVPQJOTUFA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.289275219 (498h6m18.160358254s) [noise=true , address=145.79.227.5:4223 , piece_id=DYJAX6QEQ4WQIS27QTGL2BIUIRHZGDSGYAAXYHRC5P4G5JJ6SNUQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.291396248 (498h6m18.162479293s) [noise=true , address=145.79.227.11:4730 , piece_id=6TMUM6RD3DW5LZW6FZBZ32VTGS73US66GK7WTMG6I2NU7PGSUZ5A, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.295015770 (498h6m18.166098805s) [noise=true , address=145.79.227.42:4121 , piece_id=WKT4RMHTYT6C2REQEK2H66CLIBAHU2YSTC2VWAIVUE3HTHERJZYQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.302430649 (498h6m18.173513694s) [noise=true , address=145.79.227.3:4151 , piece_id=E3COLRH263KGXAC443GE2N5E5RBV7J5QHMTXFGPUQDR2UODD7JRQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.622465599 (498h6m18.493548644s) [noise=true , address=45.91.116.11:4120 , piece_id=HCFRGCA3J3A3NAUI67B337CIK2D3PLX4IEHRWKAJDPWNZYH43BJQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.796939017 (498h6m18.668022052s) [noise=true , address=45.91.116.8:4046 , piece_id=YXCXBZBG74ZMCIUNP2JTT2E2SS233NFV44PUVS2TASJIJSJPQVEQ, seg_pos={0 0}, upload=437174]: Done uploading piece. err: <nil> Dec 26 09:35:18.796958463 (498h6m18.668041498s) [seg_pos={0 0}, upload=437174]: Segment reached optimal threshold of 13 pieces. Dec 26 09:35:18.797131523 (498h6m18.668214558s) [noise=true , address=77.90.31.21:4880 , piece_id=QGUUHNSH4BNIWDAYY7GIPGJ77ZNPKU573RTLDCCB5BLTJ3VLIHXA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:38320->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:38320->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38320->77.90.31.21:4880: use of closed network connection Dec 26 09:35:18.797137575 (498h6m18.66822061s) [noise=true , address=77.90.31.21:4880 , piece_id=QGUUHNSH4BNIWDAYY7GIPGJ77ZNPKU573RTLDCCB5BLTJ3VLIHXA, seg_pos={0 0}, upload=437174]: Long tail cancellation due to optimal threshold Dec 26 09:35:18.797323549 (498h6m18.668406594s) [noise=true , address=77.90.31.16:4735 , piece_id=CYYL53C4CVIGLFVYGXEC2YCV4UDVO2ZUTKCGLT3Z7K64ORTTJNJA, seg_pos={0 0}, upload=437174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:59758->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:59758->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59758->77.90.31.16:4735: use of closed network connection Dec 26 09:35:18.797345491 (498h6m18.668428556s) [noise=true , address=77.90.31.16:4735 , piece_id=CYYL53C4CVIGLFVYGXEC2YCV4UDVO2ZUTKCGLT3Z7K64ORTTJNJA, seg_pos={0 0}, upload=437174]: Long tail cancellation due to optimal threshold Dec 26 09:35:18.797379085 (498h6m18.66846218s) [noise=true , address=45.91.116.7:4009 , piece_id=3B6VMLZ22PDZOX67IXYRNGJSXEC2MCE25OQG5Y5WXC62SYBQTW2Q, seg_pos={0 0}, upload=437174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:57640->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:57640->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57640->45.91.116.7:4009: use of closed network connection Dec 26 09:35:18.797390957 (498h6m18.668474002s) [noise=true , address=45.91.116.7:4009 , piece_id=3B6VMLZ22PDZOX67IXYRNGJSXEC2MCE25OQG5Y5WXC62SYBQTW2Q, seg_pos={0 0}, upload=437174]: Long tail cancellation due to optimal threshold Dec 26 09:35:18.797447504 (498h6m18.668530539s) [seg_pos={0 0}, upload=437174]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:35:18.797552184 (498h6m18.668635219s) [upload=437174]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:35:18.797557735 (498h6m18.66864078s) [upload=437174]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:35:19.099664237 (498h6m18.970747272s) [upload=437174]: Done uploading segments... Dec 26 09:35:19.099752113 (498h6m18.970835158s) [upload=437174]: Upload finished. err: <nil> Duration: 30.003738103s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...........X....XXX..X.......X.....XX.XX.XXX.....X..X.....X.] duration: 15.511924923s trace: https://jaeger.dev.storj.io/trace/1ae7b019e9a95e37 FAIL (2025-12-26 09:34:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 09:33:14.680720494 (498h4m14.551803529s) [upload=437144]: Starting upload Dec 26 09:33:14.680777191 (498h4m14.551860236s) [upload=437144]: Done starting upload Dec 26 09:33:14.680875078 (498h4m14.551958123s) [upload=437144]: Uploading segments... Dec 26 09:33:14.681005947 (498h4m14.552088992s) [upload=437144]: Got next segment. Inline: false Dec 26 09:33:14.681072664 (498h4m14.552155699s) [upload=437144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:33:14.681079407 (498h4m14.552162442s) [upload=437144]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:33:14.925198341 (498h4m14.796281376s) [seg_pos={0 0}, upload=437144]: Begin upload segment... Dec 26 09:33:14.925261050 (498h4m14.796344085s) [seg_pos={0 0}, upload=437144]: Done begin upload segment. Dec 26 09:33:14.925298592 (498h4m14.796381627s) [noise=true , address=145.79.227.16:4909 , piece_id=BS3HUXORR2HA4HRYGVGXUG7FVCTDB44GD6376QR7BKC76HFVUHEA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.925303902 (498h4m14.796386937s) [noise=true , address=45.91.116.6:4975 , piece_id=PXK4EAL64EAQGMSQVAP454GNUFT2DWHVFY6FIISSQJ6N345KYX4A, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.925355841 (498h4m14.796438876s) [noise=true , address=145.79.227.43:4158 , piece_id=K4FB3K5JVED3YRLZGVEN3NTFIY7ERXNKY6BLORJ24RPOEMRBXKMA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.925389034 (498h4m14.796472069s) [noise=true , address=45.91.116.10:4084 , piece_id=LISJ7UU7XAL5VIRHPCJLKTN5WAOB2CNZWWINGXRYHINEHD7R3XOA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.925872816 (498h4m14.796955861s) [noise=true , address=145.79.227.36:4517 , piece_id=RZEUJ37PLVW2DRUK4OY5FRYV5HYGHEPOHRJXBIBKAGTEVAHZIHOA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.928051163 (498h4m14.799134208s) [noise=true , address=45.91.116.5:4299 , piece_id=6BXKHTBHENB5BOORFJJGGMZBIP6S7WQLBTTUTRHDILHUE2ZY2JQA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.928467676 (498h4m14.799550711s) [noise=true , address=145.79.227.22:4511 , piece_id=T2LKM53FXCPZSXFYZ66RXNWHMWH3XML7MVGKTV5HG25TR5PHOOAQ, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.929988532 (498h4m14.801071567s) [noise=true , address=77.90.31.19:4844 , piece_id=EARFNIEZZ4HUEJUWJIFBD5U2M3NBEE2VJ7ZPJTQJ44PU5L3SRLEQ, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.931318745 (498h4m14.80240178s) [noise=true , address=145.79.227.51:4409 , piece_id=ZQOWDCDC5NEPJAQO6GAI6N6RG6AAXICJEYYBJFU655PDEWYA7CLQ, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.932494093 (498h4m14.803577129s) [noise=true , address=145.79.227.3:4151 , piece_id=3HUN7UTT7OOSTUS3I2ZZMR5X32LILZFFDC5NO2ZVPUGBHPG75GYA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.933354952 (498h4m14.804437997s) [noise=true , address=145.79.227.48:4301 , piece_id=74HZ7BWHIOERWNPU6PQGN2LV7MIHO5JPHXVFG7OJZN25UO2GIDTA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.934731283 (498h4m14.805814318s) [noise=true , address=145.79.227.19:4077 , piece_id=M2ZMEVLY5FEH5N4OZ46LSZFC4JCJOCP5JAAF65G4IC5ZAMVQ55IA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.935061001 (498h4m14.806144046s) [noise=true , address=145.79.227.24:4583 , piece_id=ODW3UBJF2RSAFOPMXSM3HGRCFR4HM3OHYXR3EBEE46GR6JBKMEAA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.936248512 (498h4m14.807331557s) [noise=true , address=145.79.227.4:4185 , piece_id=XK3TYVVP4DFGIRZNQGXYHIXE3ZYSF4PQHO7LJE5IZ55YRYVIM2HQ, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.936259353 (498h4m14.807342398s) [noise=true , address=145.79.227.49:4339 , piece_id=QVUF4QOI64FHTFDKT73DVZ2ZWAR5WHKLJZR4KPZXK4OXH5YMOFAQ, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.939702780 (498h4m14.810785815s) [noise=true , address=145.79.227.40:4051 , piece_id=ZOXMKVFM2S6PPAZI6BYFEWWII2YSCS7DGWWPERJ27XV7AVNW7A5A, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:16.580367713 (498h4m16.451450748s) [upload=437144]: Next returned nil segment Dec 26 09:33:16.580386599 (498h4m16.451469634s) [upload=437144]: Waiting for error group managing segments... Dec 26 09:33:16.928497811 (498h4m16.799580856s) [noise=true , address=145.79.227.48:4301 , piece_id=74HZ7BWHIOERWNPU6PQGN2LV7MIHO5JPHXVFG7OJZN25UO2GIDTA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.929162727 (498h4m16.800245773s) [noise=true , address=145.79.227.19:4077 , piece_id=M2ZMEVLY5FEH5N4OZ46LSZFC4JCJOCP5JAAF65G4IC5ZAMVQ55IA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.934996787 (498h4m16.806079832s) [noise=true , address=145.79.227.16:4909 , piece_id=BS3HUXORR2HA4HRYGVGXUG7FVCTDB44GD6376QR7BKC76HFVUHEA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.938366313 (498h4m16.809449358s) [noise=true , address=145.79.227.36:4517 , piece_id=RZEUJ37PLVW2DRUK4OY5FRYV5HYGHEPOHRJXBIBKAGTEVAHZIHOA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.939059462 (498h4m16.810142507s) [noise=true , address=145.79.227.43:4158 , piece_id=K4FB3K5JVED3YRLZGVEN3NTFIY7ERXNKY6BLORJ24RPOEMRBXKMA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.939807607 (498h4m16.810890652s) [noise=true , address=145.79.227.51:4409 , piece_id=ZQOWDCDC5NEPJAQO6GAI6N6RG6AAXICJEYYBJFU655PDEWYA7CLQ, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.941331630 (498h4m16.812414665s) [noise=true , address=145.79.227.4:4185 , piece_id=XK3TYVVP4DFGIRZNQGXYHIXE3ZYSF4PQHO7LJE5IZ55YRYVIM2HQ, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.958341579 (498h4m16.829424624s) [noise=true , address=145.79.227.22:4511 , piece_id=T2LKM53FXCPZSXFYZ66RXNWHMWH3XML7MVGKTV5HG25TR5PHOOAQ, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.968182648 (498h4m16.839265683s) [noise=true , address=145.79.227.49:4339 , piece_id=QVUF4QOI64FHTFDKT73DVZ2ZWAR5WHKLJZR4KPZXK4OXH5YMOFAQ, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.968578862 (498h4m16.839661897s) [noise=true , address=145.79.227.24:4583 , piece_id=ODW3UBJF2RSAFOPMXSM3HGRCFR4HM3OHYXR3EBEE46GR6JBKMEAA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:17.083971951 (498h4m16.955054996s) [noise=true , address=145.79.227.3:4151 , piece_id=3HUN7UTT7OOSTUS3I2ZZMR5X32LILZFFDC5NO2ZVPUGBHPG75GYA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:17.195657729 (498h4m17.066740774s) [noise=true , address=45.91.116.6:4975 , piece_id=PXK4EAL64EAQGMSQVAP454GNUFT2DWHVFY6FIISSQJ6N345KYX4A, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:17.282992739 (498h4m17.154075775s) [noise=true , address=45.91.116.5:4299 , piece_id=6BXKHTBHENB5BOORFJJGGMZBIP6S7WQLBTTUTRHDILHUE2ZY2JQA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:17.283011806 (498h4m17.154094851s) [seg_pos={0 0}, upload=437144]: Segment reached optimal threshold of 13 pieces. Dec 26 09:33:17.283253667 (498h4m17.154336702s) [noise=true , address=145.79.227.40:4051 , piece_id=ZOXMKVFM2S6PPAZI6BYFEWWII2YSCS7DGWWPERJ27XV7AVNW7A5A, seg_pos={0 0}, upload=437144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 09:33:17.283261031 (498h4m17.154344066s) [noise=true , address=145.79.227.40:4051 , piece_id=ZOXMKVFM2S6PPAZI6BYFEWWII2YSCS7DGWWPERJ27XV7AVNW7A5A, seg_pos={0 0}, upload=437144]: Long tail cancellation due to optimal threshold Dec 26 09:33:17.283496960 (498h4m17.154579995s) [noise=true , address=45.91.116.10:4084 , piece_id=LISJ7UU7XAL5VIRHPCJLKTN5WAOB2CNZWWINGXRYHINEHD7R3XOA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 09:33:17.283503042 (498h4m17.154586077s) [noise=true , address=45.91.116.10:4084 , piece_id=LISJ7UU7XAL5VIRHPCJLKTN5WAOB2CNZWWINGXRYHINEHD7R3XOA, seg_pos={0 0}, upload=437144]: Long tail cancellation due to optimal threshold Dec 26 09:33:17.283624883 (498h4m17.154707918s) [noise=true , address=77.90.31.19:4844 , piece_id=EARFNIEZZ4HUEJUWJIFBD5U2M3NBEE2VJ7ZPJTQJ44PU5L3SRLEQ, seg_pos={0 0}, upload=437144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 26 09:33:17.283630094 (498h4m17.154713129s) [noise=true , address=77.90.31.19:4844 , piece_id=EARFNIEZZ4HUEJUWJIFBD5U2M3NBEE2VJ7ZPJTQJ44PU5L3SRLEQ, seg_pos={0 0}, upload=437144]: Long tail cancellation due to optimal threshold Dec 26 09:33:17.283643549 (498h4m17.154726594s) [seg_pos={0 0}, upload=437144]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:33:17.283708623 (498h4m17.154791658s) [upload=437144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:33:17.283713322 (498h4m17.154796357s) [upload=437144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:33:17.569219914 (498h4m17.440302949s) [upload=437144]: Done uploading segments... Dec 26 09:33:17.569242868 (498h4m17.440325913s) [upload=437144]: Upload finished. err: <nil> Duration: 15.511951484s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [............X....XXX..X.......X.....XX.XX.XXX.....X..X.....X] duration: 30.002629404s trace: https://jaeger.dev.storj.io/trace/1ae7b019e9a95e37 FAIL (2025-12-26 09:33:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 09:33:14.680720494 (498h4m14.551803529s) [upload=437144]: Starting upload Dec 26 09:33:14.680777191 (498h4m14.551860236s) [upload=437144]: Done starting upload Dec 26 09:33:14.680875078 (498h4m14.551958123s) [upload=437144]: Uploading segments... Dec 26 09:33:14.681005947 (498h4m14.552088992s) [upload=437144]: Got next segment. Inline: false Dec 26 09:33:14.681072664 (498h4m14.552155699s) [upload=437144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:33:14.681079407 (498h4m14.552162442s) [upload=437144]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:33:14.925198341 (498h4m14.796281376s) [seg_pos={0 0}, upload=437144]: Begin upload segment... Dec 26 09:33:14.925261050 (498h4m14.796344085s) [seg_pos={0 0}, upload=437144]: Done begin upload segment. Dec 26 09:33:14.925298592 (498h4m14.796381627s) [noise=true , address=145.79.227.16:4909 , piece_id=BS3HUXORR2HA4HRYGVGXUG7FVCTDB44GD6376QR7BKC76HFVUHEA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.925303902 (498h4m14.796386937s) [noise=true , address=45.91.116.6:4975 , piece_id=PXK4EAL64EAQGMSQVAP454GNUFT2DWHVFY6FIISSQJ6N345KYX4A, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.925355841 (498h4m14.796438876s) [noise=true , address=145.79.227.43:4158 , piece_id=K4FB3K5JVED3YRLZGVEN3NTFIY7ERXNKY6BLORJ24RPOEMRBXKMA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.925389034 (498h4m14.796472069s) [noise=true , address=45.91.116.10:4084 , piece_id=LISJ7UU7XAL5VIRHPCJLKTN5WAOB2CNZWWINGXRYHINEHD7R3XOA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.925872816 (498h4m14.796955861s) [noise=true , address=145.79.227.36:4517 , piece_id=RZEUJ37PLVW2DRUK4OY5FRYV5HYGHEPOHRJXBIBKAGTEVAHZIHOA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.928051163 (498h4m14.799134208s) [noise=true , address=45.91.116.5:4299 , piece_id=6BXKHTBHENB5BOORFJJGGMZBIP6S7WQLBTTUTRHDILHUE2ZY2JQA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.928467676 (498h4m14.799550711s) [noise=true , address=145.79.227.22:4511 , piece_id=T2LKM53FXCPZSXFYZ66RXNWHMWH3XML7MVGKTV5HG25TR5PHOOAQ, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.929988532 (498h4m14.801071567s) [noise=true , address=77.90.31.19:4844 , piece_id=EARFNIEZZ4HUEJUWJIFBD5U2M3NBEE2VJ7ZPJTQJ44PU5L3SRLEQ, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.931318745 (498h4m14.80240178s) [noise=true , address=145.79.227.51:4409 , piece_id=ZQOWDCDC5NEPJAQO6GAI6N6RG6AAXICJEYYBJFU655PDEWYA7CLQ, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.932494093 (498h4m14.803577129s) [noise=true , address=145.79.227.3:4151 , piece_id=3HUN7UTT7OOSTUS3I2ZZMR5X32LILZFFDC5NO2ZVPUGBHPG75GYA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.933354952 (498h4m14.804437997s) [noise=true , address=145.79.227.48:4301 , piece_id=74HZ7BWHIOERWNPU6PQGN2LV7MIHO5JPHXVFG7OJZN25UO2GIDTA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.934731283 (498h4m14.805814318s) [noise=true , address=145.79.227.19:4077 , piece_id=M2ZMEVLY5FEH5N4OZ46LSZFC4JCJOCP5JAAF65G4IC5ZAMVQ55IA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.935061001 (498h4m14.806144046s) [noise=true , address=145.79.227.24:4583 , piece_id=ODW3UBJF2RSAFOPMXSM3HGRCFR4HM3OHYXR3EBEE46GR6JBKMEAA, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.936248512 (498h4m14.807331557s) [noise=true , address=145.79.227.4:4185 , piece_id=XK3TYVVP4DFGIRZNQGXYHIXE3ZYSF4PQHO7LJE5IZ55YRYVIM2HQ, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.936259353 (498h4m14.807342398s) [noise=true , address=145.79.227.49:4339 , piece_id=QVUF4QOI64FHTFDKT73DVZ2ZWAR5WHKLJZR4KPZXK4OXH5YMOFAQ, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:14.939702780 (498h4m14.810785815s) [noise=true , address=145.79.227.40:4051 , piece_id=ZOXMKVFM2S6PPAZI6BYFEWWII2YSCS7DGWWPERJ27XV7AVNW7A5A, seg_pos={0 0}, upload=437144]: Uploading piece... Dec 26 09:33:16.580367713 (498h4m16.451450748s) [upload=437144]: Next returned nil segment Dec 26 09:33:16.580386599 (498h4m16.451469634s) [upload=437144]: Waiting for error group managing segments... Dec 26 09:33:16.928497811 (498h4m16.799580856s) [noise=true , address=145.79.227.48:4301 , piece_id=74HZ7BWHIOERWNPU6PQGN2LV7MIHO5JPHXVFG7OJZN25UO2GIDTA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.929162727 (498h4m16.800245773s) [noise=true , address=145.79.227.19:4077 , piece_id=M2ZMEVLY5FEH5N4OZ46LSZFC4JCJOCP5JAAF65G4IC5ZAMVQ55IA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.934996787 (498h4m16.806079832s) [noise=true , address=145.79.227.16:4909 , piece_id=BS3HUXORR2HA4HRYGVGXUG7FVCTDB44GD6376QR7BKC76HFVUHEA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.938366313 (498h4m16.809449358s) [noise=true , address=145.79.227.36:4517 , piece_id=RZEUJ37PLVW2DRUK4OY5FRYV5HYGHEPOHRJXBIBKAGTEVAHZIHOA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.939059462 (498h4m16.810142507s) [noise=true , address=145.79.227.43:4158 , piece_id=K4FB3K5JVED3YRLZGVEN3NTFIY7ERXNKY6BLORJ24RPOEMRBXKMA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.939807607 (498h4m16.810890652s) [noise=true , address=145.79.227.51:4409 , piece_id=ZQOWDCDC5NEPJAQO6GAI6N6RG6AAXICJEYYBJFU655PDEWYA7CLQ, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.941331630 (498h4m16.812414665s) [noise=true , address=145.79.227.4:4185 , piece_id=XK3TYVVP4DFGIRZNQGXYHIXE3ZYSF4PQHO7LJE5IZ55YRYVIM2HQ, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.958341579 (498h4m16.829424624s) [noise=true , address=145.79.227.22:4511 , piece_id=T2LKM53FXCPZSXFYZ66RXNWHMWH3XML7MVGKTV5HG25TR5PHOOAQ, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.968182648 (498h4m16.839265683s) [noise=true , address=145.79.227.49:4339 , piece_id=QVUF4QOI64FHTFDKT73DVZ2ZWAR5WHKLJZR4KPZXK4OXH5YMOFAQ, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:16.968578862 (498h4m16.839661897s) [noise=true , address=145.79.227.24:4583 , piece_id=ODW3UBJF2RSAFOPMXSM3HGRCFR4HM3OHYXR3EBEE46GR6JBKMEAA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:17.083971951 (498h4m16.955054996s) [noise=true , address=145.79.227.3:4151 , piece_id=3HUN7UTT7OOSTUS3I2ZZMR5X32LILZFFDC5NO2ZVPUGBHPG75GYA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:17.195657729 (498h4m17.066740774s) [noise=true , address=45.91.116.6:4975 , piece_id=PXK4EAL64EAQGMSQVAP454GNUFT2DWHVFY6FIISSQJ6N345KYX4A, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:17.282992739 (498h4m17.154075775s) [noise=true , address=45.91.116.5:4299 , piece_id=6BXKHTBHENB5BOORFJJGGMZBIP6S7WQLBTTUTRHDILHUE2ZY2JQA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: <nil> Dec 26 09:33:17.283011806 (498h4m17.154094851s) [seg_pos={0 0}, upload=437144]: Segment reached optimal threshold of 13 pieces. Dec 26 09:33:17.283253667 (498h4m17.154336702s) [noise=true , address=145.79.227.40:4051 , piece_id=ZOXMKVFM2S6PPAZI6BYFEWWII2YSCS7DGWWPERJ27XV7AVNW7A5A, seg_pos={0 0}, upload=437144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 09:33:17.283261031 (498h4m17.154344066s) [noise=true , address=145.79.227.40:4051 , piece_id=ZOXMKVFM2S6PPAZI6BYFEWWII2YSCS7DGWWPERJ27XV7AVNW7A5A, seg_pos={0 0}, upload=437144]: Long tail cancellation due to optimal threshold Dec 26 09:33:17.283496960 (498h4m17.154579995s) [noise=true , address=45.91.116.10:4084 , piece_id=LISJ7UU7XAL5VIRHPCJLKTN5WAOB2CNZWWINGXRYHINEHD7R3XOA, seg_pos={0 0}, upload=437144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 09:33:17.283503042 (498h4m17.154586077s) [noise=true , address=45.91.116.10:4084 , piece_id=LISJ7UU7XAL5VIRHPCJLKTN5WAOB2CNZWWINGXRYHINEHD7R3XOA, seg_pos={0 0}, upload=437144]: Long tail cancellation due to optimal threshold Dec 26 09:33:17.283624883 (498h4m17.154707918s) [noise=true , address=77.90.31.19:4844 , piece_id=EARFNIEZZ4HUEJUWJIFBD5U2M3NBEE2VJ7ZPJTQJ44PU5L3SRLEQ, seg_pos={0 0}, upload=437144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 26 09:33:17.283630094 (498h4m17.154713129s) [noise=true , address=77.90.31.19:4844 , piece_id=EARFNIEZZ4HUEJUWJIFBD5U2M3NBEE2VJ7ZPJTQJ44PU5L3SRLEQ, seg_pos={0 0}, upload=437144]: Long tail cancellation due to optimal threshold Dec 26 09:33:17.283643549 (498h4m17.154726594s) [seg_pos={0 0}, upload=437144]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:33:17.283708623 (498h4m17.154791658s) [upload=437144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:33:17.283713322 (498h4m17.154796357s) [upload=437144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:33:17.569219914 (498h4m17.440302949s) [upload=437144]: Done uploading segments... Dec 26 09:33:17.569242868 (498h4m17.440325913s) [upload=437144]: Upload finished. err: <nil> Duration: 30.002654272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............X....XXX..X.......X.....XX.XX.XXX.....X..X.....] duration: 5.975974036s trace: https://jaeger.dev.storj.io/trace/319e22bd076acae8 FAIL (2025-12-26 09:32:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ) context deadline exceeded ======LOGS====== oad=437071]: Uploading segments... Dec 26 09:28:08.418955570 (497h59m8.290038605s) [upload=437071]: Got next segment. Inline: false Dec 26 09:28:08.419031294 (497h59m8.290114329s) [upload=437071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:28:08.419036804 (497h59m8.290119839s) [upload=437071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:28:08.663968175 (497h59m8.53505126s) [seg_pos={0 0}, upload=437071]: Begin upload segment... Dec 26 09:28:08.664026606 (497h59m8.535109642s) [seg_pos={0 0}, upload=437071]: Done begin upload segment. Dec 26 09:28:08.664068326 (497h59m8.535151371s) [noise=true , address=145.79.227.52:4447 , piece_id=QPOPFXJLXURCKFPXWSE7HQPXEZRV3XZ5R74BP55C3J6E67U7MK6Q, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.664145823 (497h59m8.535228868s) [noise=true , address=145.79.227.11:4731 , piece_id=XJPXNJQHVIYLEPV22DZYULXUVMWHEN5WSGJLRLN6BPC2XZB4MOCQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.664921371 (497h59m8.536004416s) [noise=true , address=145.79.227.4:4186 , piece_id=7O3BVVNANVOCSC4OZY5Q3YMXGXYE4OUEBCPVT4PYMMXVCHPRYZHA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665214528 (497h59m8.536297573s) [noise=true , address=45.91.116.10:4081 , piece_id=DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665614420 (497h59m8.536697455s) [noise=true , address=45.91.116.8:4046 , piece_id=JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665993302 (497h59m8.537076337s) [noise=true , address=45.91.116.12:4226 , piece_id=O372B7PCAKW4AUAHRYBHBVGVLE2W734JCS6UNXEZJQHLC3K6DDZQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666040852 (497h59m8.537123887s) [noise=true , address=145.79.227.42:4122 , piece_id=6K4GMP2A4VAOO3Y6LPALBPXBVWLA54AR4SM5MA4YTETBBY2B752A, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666405606 (497h59m8.537488651s) [noise=true , address=145.79.227.50:4374 , piece_id=CVOZ4UIWRNBZAIN5RKEKZ32ICGZFD34TVPLDYMVJW435V3RZC6CA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666702762 (497h59m8.537785798s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666995319 (497h59m8.538078364s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667420760 (497h59m8.538503795s) [noise=true , address=145.79.227.38:4979 , piece_id=BVH2BXNSN2G3MNESWFVFKI52KS72DDB6TWTJPDTFNIDODIEJXGSA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667791636 (497h59m8.538874671s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667930320 (497h59m8.539013365s) [noise=true , address=145.79.227.46:4603 , piece_id=YFLTIC3TDVZL5VSEVPTZA4OJCPNODT3ZQ3QVLEK2T7GBHM7WAKTQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.668574898 (497h59m8.539657933s) [noise=true , address=145.79.227.45:4231 , piece_id=WOXIAN5R7RTWOUIFD7GWRPRQHNUA6S6BUAYPH4STJ4LV2VAEGXFA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.668945613 (497h59m8.540028658s) [noise=true , address=145.79.227.39:4014 , piece_id=Y5AZV5RYKECLQWNIBC7ETIMM274GE2AV3M5I3MNJA5PTWCCHA5HA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.669338902 (497h59m8.540421937s) [noise=true , address=45.91.116.14:4189 , piece_id=XFLQI3GPFZHLMLEFBIMXMA2SC6P7VWVNIKYEWLHOKDUZQW5KFHMA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:10.290945294 (497h59m10.16202835s) [upload=437071]: Next returned nil segment Dec 26 09:28:10.290958509 (497h59m10.162041554s) [upload=437071]: Waiting for error group managing segments... Dec 26 09:28:10.678285896 (497h59m10.549368931s) [noise=true , address=145.79.227.50:4374 , piece_id=CVOZ4UIWRNBZAIN5RKEKZ32ICGZFD34TVPLDYMVJW435V3RZC6CA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.681122548 (497h59m10.552205583s) [noise=true , address=145.79.227.42:4122 , piece_id=6K4GMP2A4VAOO3Y6LPALBPXBVWLA54AR4SM5MA4YTETBBY2B752A, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.683302087 (497h59m10.554385132s) [noise=true , address=145.79.227.4:4186 , piece_id=7O3BVVNANVOCSC4OZY5Q3YMXGXYE4OUEBCPVT4PYMMXVCHPRYZHA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.683990328 (497h59m10.555073363s) [noise=true , address=145.79.227.52:4447 , piece_id=QPOPFXJLXURCKFPXWSE7HQPXEZRV3XZ5R74BP55C3J6E67U7MK6Q, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.693692112 (497h59m10.564775157s) [noise=true , address=145.79.227.39:4014 , piece_id=Y5AZV5RYKECLQWNIBC7ETIMM274GE2AV3M5I3MNJA5PTWCCHA5HA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.696002942 (497h59m10.567085987s) [noise=true , address=145.79.227.45:4231 , piece_id=WOXIAN5R7RTWOUIFD7GWRPRQHNUA6S6BUAYPH4STJ4LV2VAEGXFA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.697317195 (497h59m10.56840023s) [noise=true , address=145.79.227.11:4731 , piece_id=XJPXNJQHVIYLEPV22DZYULXUVMWHEN5WSGJLRLN6BPC2XZB4MOCQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.705818693 (497h59m10.576901738s) [noise=true , address=145.79.227.38:4979 , piece_id=BVH2BXNSN2G3MNESWFVFKI52KS72DDB6TWTJPDTFNIDODIEJXGSA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.713470103 (497h59m10.584553148s) [noise=true , address=145.79.227.46:4603 , piece_id=YFLTIC3TDVZL5VSEVPTZA4OJCPNODT3ZQ3QVLEK2T7GBHM7WAKTQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.955672247 (497h59m10.826755292s) [noise=true , address=45.91.116.14:4189 , piece_id=XFLQI3GPFZHLMLEFBIMXMA2SC6P7VWVNIKYEWLHOKDUZQW5KFHMA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.992053222 (497h59m10.863136257s) [noise=true , address=45.91.116.12:4226 , piece_id=O372B7PCAKW4AUAHRYBHBVGVLE2W734JCS6UNXEZJQHLC3K6DDZQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.024551975 (497h59m10.89563502s) [noise=true , address=45.91.116.8:4046 , piece_id=JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.031430964 (497h59m10.902514009s) [noise=true , address=45.91.116.10:4081 , piece_id=DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.031476941 (497h59m10.902559987s) [seg_pos={0 0}, upload=437071]: Segment reached optimal threshold of 13 pieces. Dec 26 09:28:11.031753057 (497h59m10.902836102s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection Dec 26 09:28:11.031803594 (497h59m10.902886659s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032208154 (497h59m10.903291189s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 09:28:11.032217091 (497h59m10.903300126s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032346407 (497h59m10.903429452s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection Dec 26 09:28:11.032354913 (497h59m10.903437958s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032372537 (497h59m10.903455582s) [seg_pos={0 0}, upload=437071]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:28:11.032470764 (497h59m10.903553799s) [upload=437071]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:28:11.032477387 (497h59m10.903560432s) [upload=437071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:28:11.319582514 (497h59m11.190665559s) [upload=437071]: Done uploading segments... Dec 26 09:28:11.319601119 (497h59m11.190684154s) [upload=437071]: Upload finished. err: <nil> Duration: 5.976021757s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..............X....XXX..X.......X.....XX.XX.XXX.....X..X....] duration: 6.023293961s trace: https://jaeger.dev.storj.io/trace/319e22bd076acae8 FAIL (2025-12-26 09:31:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ) context deadline exceeded ======LOGS====== oad=437071]: Uploading segments... Dec 26 09:28:08.418955570 (497h59m8.290038605s) [upload=437071]: Got next segment. Inline: false Dec 26 09:28:08.419031294 (497h59m8.290114329s) [upload=437071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:28:08.419036804 (497h59m8.290119839s) [upload=437071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:28:08.663968175 (497h59m8.53505126s) [seg_pos={0 0}, upload=437071]: Begin upload segment... Dec 26 09:28:08.664026606 (497h59m8.535109642s) [seg_pos={0 0}, upload=437071]: Done begin upload segment. Dec 26 09:28:08.664068326 (497h59m8.535151371s) [noise=true , address=145.79.227.52:4447 , piece_id=QPOPFXJLXURCKFPXWSE7HQPXEZRV3XZ5R74BP55C3J6E67U7MK6Q, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.664145823 (497h59m8.535228868s) [noise=true , address=145.79.227.11:4731 , piece_id=XJPXNJQHVIYLEPV22DZYULXUVMWHEN5WSGJLRLN6BPC2XZB4MOCQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.664921371 (497h59m8.536004416s) [noise=true , address=145.79.227.4:4186 , piece_id=7O3BVVNANVOCSC4OZY5Q3YMXGXYE4OUEBCPVT4PYMMXVCHPRYZHA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665214528 (497h59m8.536297573s) [noise=true , address=45.91.116.10:4081 , piece_id=DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665614420 (497h59m8.536697455s) [noise=true , address=45.91.116.8:4046 , piece_id=JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665993302 (497h59m8.537076337s) [noise=true , address=45.91.116.12:4226 , piece_id=O372B7PCAKW4AUAHRYBHBVGVLE2W734JCS6UNXEZJQHLC3K6DDZQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666040852 (497h59m8.537123887s) [noise=true , address=145.79.227.42:4122 , piece_id=6K4GMP2A4VAOO3Y6LPALBPXBVWLA54AR4SM5MA4YTETBBY2B752A, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666405606 (497h59m8.537488651s) [noise=true , address=145.79.227.50:4374 , piece_id=CVOZ4UIWRNBZAIN5RKEKZ32ICGZFD34TVPLDYMVJW435V3RZC6CA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666702762 (497h59m8.537785798s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666995319 (497h59m8.538078364s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667420760 (497h59m8.538503795s) [noise=true , address=145.79.227.38:4979 , piece_id=BVH2BXNSN2G3MNESWFVFKI52KS72DDB6TWTJPDTFNIDODIEJXGSA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667791636 (497h59m8.538874671s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667930320 (497h59m8.539013365s) [noise=true , address=145.79.227.46:4603 , piece_id=YFLTIC3TDVZL5VSEVPTZA4OJCPNODT3ZQ3QVLEK2T7GBHM7WAKTQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.668574898 (497h59m8.539657933s) [noise=true , address=145.79.227.45:4231 , piece_id=WOXIAN5R7RTWOUIFD7GWRPRQHNUA6S6BUAYPH4STJ4LV2VAEGXFA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.668945613 (497h59m8.540028658s) [noise=true , address=145.79.227.39:4014 , piece_id=Y5AZV5RYKECLQWNIBC7ETIMM274GE2AV3M5I3MNJA5PTWCCHA5HA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.669338902 (497h59m8.540421937s) [noise=true , address=45.91.116.14:4189 , piece_id=XFLQI3GPFZHLMLEFBIMXMA2SC6P7VWVNIKYEWLHOKDUZQW5KFHMA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:10.290945294 (497h59m10.16202835s) [upload=437071]: Next returned nil segment Dec 26 09:28:10.290958509 (497h59m10.162041554s) [upload=437071]: Waiting for error group managing segments... Dec 26 09:28:10.678285896 (497h59m10.549368931s) [noise=true , address=145.79.227.50:4374 , piece_id=CVOZ4UIWRNBZAIN5RKEKZ32ICGZFD34TVPLDYMVJW435V3RZC6CA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.681122548 (497h59m10.552205583s) [noise=true , address=145.79.227.42:4122 , piece_id=6K4GMP2A4VAOO3Y6LPALBPXBVWLA54AR4SM5MA4YTETBBY2B752A, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.683302087 (497h59m10.554385132s) [noise=true , address=145.79.227.4:4186 , piece_id=7O3BVVNANVOCSC4OZY5Q3YMXGXYE4OUEBCPVT4PYMMXVCHPRYZHA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.683990328 (497h59m10.555073363s) [noise=true , address=145.79.227.52:4447 , piece_id=QPOPFXJLXURCKFPXWSE7HQPXEZRV3XZ5R74BP55C3J6E67U7MK6Q, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.693692112 (497h59m10.564775157s) [noise=true , address=145.79.227.39:4014 , piece_id=Y5AZV5RYKECLQWNIBC7ETIMM274GE2AV3M5I3MNJA5PTWCCHA5HA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.696002942 (497h59m10.567085987s) [noise=true , address=145.79.227.45:4231 , piece_id=WOXIAN5R7RTWOUIFD7GWRPRQHNUA6S6BUAYPH4STJ4LV2VAEGXFA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.697317195 (497h59m10.56840023s) [noise=true , address=145.79.227.11:4731 , piece_id=XJPXNJQHVIYLEPV22DZYULXUVMWHEN5WSGJLRLN6BPC2XZB4MOCQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.705818693 (497h59m10.576901738s) [noise=true , address=145.79.227.38:4979 , piece_id=BVH2BXNSN2G3MNESWFVFKI52KS72DDB6TWTJPDTFNIDODIEJXGSA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.713470103 (497h59m10.584553148s) [noise=true , address=145.79.227.46:4603 , piece_id=YFLTIC3TDVZL5VSEVPTZA4OJCPNODT3ZQ3QVLEK2T7GBHM7WAKTQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.955672247 (497h59m10.826755292s) [noise=true , address=45.91.116.14:4189 , piece_id=XFLQI3GPFZHLMLEFBIMXMA2SC6P7VWVNIKYEWLHOKDUZQW5KFHMA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.992053222 (497h59m10.863136257s) [noise=true , address=45.91.116.12:4226 , piece_id=O372B7PCAKW4AUAHRYBHBVGVLE2W734JCS6UNXEZJQHLC3K6DDZQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.024551975 (497h59m10.89563502s) [noise=true , address=45.91.116.8:4046 , piece_id=JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.031430964 (497h59m10.902514009s) [noise=true , address=45.91.116.10:4081 , piece_id=DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.031476941 (497h59m10.902559987s) [seg_pos={0 0}, upload=437071]: Segment reached optimal threshold of 13 pieces. Dec 26 09:28:11.031753057 (497h59m10.902836102s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection Dec 26 09:28:11.031803594 (497h59m10.902886659s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032208154 (497h59m10.903291189s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 09:28:11.032217091 (497h59m10.903300126s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032346407 (497h59m10.903429452s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection Dec 26 09:28:11.032354913 (497h59m10.903437958s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032372537 (497h59m10.903455582s) [seg_pos={0 0}, upload=437071]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:28:11.032470764 (497h59m10.903553799s) [upload=437071]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:28:11.032477387 (497h59m10.903560432s) [upload=437071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:28:11.319582514 (497h59m11.190665559s) [upload=437071]: Done uploading segments... Dec 26 09:28:11.319601119 (497h59m11.190684154s) [upload=437071]: Upload finished. err: <nil> Duration: 6.023310933s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...............X....XXX..X.......X.....XX.XX.XXX.....X..X...] duration: 5.726548595s trace: https://jaeger.dev.storj.io/trace/319e22bd076acae8 FAIL (2025-12-26 09:31:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ) context deadline exceeded ======LOGS====== oad=437071]: Uploading segments... Dec 26 09:28:08.418955570 (497h59m8.290038605s) [upload=437071]: Got next segment. Inline: false Dec 26 09:28:08.419031294 (497h59m8.290114329s) [upload=437071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:28:08.419036804 (497h59m8.290119839s) [upload=437071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:28:08.663968175 (497h59m8.53505126s) [seg_pos={0 0}, upload=437071]: Begin upload segment... Dec 26 09:28:08.664026606 (497h59m8.535109642s) [seg_pos={0 0}, upload=437071]: Done begin upload segment. Dec 26 09:28:08.664068326 (497h59m8.535151371s) [noise=true , address=145.79.227.52:4447 , piece_id=QPOPFXJLXURCKFPXWSE7HQPXEZRV3XZ5R74BP55C3J6E67U7MK6Q, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.664145823 (497h59m8.535228868s) [noise=true , address=145.79.227.11:4731 , piece_id=XJPXNJQHVIYLEPV22DZYULXUVMWHEN5WSGJLRLN6BPC2XZB4MOCQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.664921371 (497h59m8.536004416s) [noise=true , address=145.79.227.4:4186 , piece_id=7O3BVVNANVOCSC4OZY5Q3YMXGXYE4OUEBCPVT4PYMMXVCHPRYZHA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665214528 (497h59m8.536297573s) [noise=true , address=45.91.116.10:4081 , piece_id=DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665614420 (497h59m8.536697455s) [noise=true , address=45.91.116.8:4046 , piece_id=JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665993302 (497h59m8.537076337s) [noise=true , address=45.91.116.12:4226 , piece_id=O372B7PCAKW4AUAHRYBHBVGVLE2W734JCS6UNXEZJQHLC3K6DDZQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666040852 (497h59m8.537123887s) [noise=true , address=145.79.227.42:4122 , piece_id=6K4GMP2A4VAOO3Y6LPALBPXBVWLA54AR4SM5MA4YTETBBY2B752A, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666405606 (497h59m8.537488651s) [noise=true , address=145.79.227.50:4374 , piece_id=CVOZ4UIWRNBZAIN5RKEKZ32ICGZFD34TVPLDYMVJW435V3RZC6CA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666702762 (497h59m8.537785798s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666995319 (497h59m8.538078364s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667420760 (497h59m8.538503795s) [noise=true , address=145.79.227.38:4979 , piece_id=BVH2BXNSN2G3MNESWFVFKI52KS72DDB6TWTJPDTFNIDODIEJXGSA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667791636 (497h59m8.538874671s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667930320 (497h59m8.539013365s) [noise=true , address=145.79.227.46:4603 , piece_id=YFLTIC3TDVZL5VSEVPTZA4OJCPNODT3ZQ3QVLEK2T7GBHM7WAKTQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.668574898 (497h59m8.539657933s) [noise=true , address=145.79.227.45:4231 , piece_id=WOXIAN5R7RTWOUIFD7GWRPRQHNUA6S6BUAYPH4STJ4LV2VAEGXFA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.668945613 (497h59m8.540028658s) [noise=true , address=145.79.227.39:4014 , piece_id=Y5AZV5RYKECLQWNIBC7ETIMM274GE2AV3M5I3MNJA5PTWCCHA5HA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.669338902 (497h59m8.540421937s) [noise=true , address=45.91.116.14:4189 , piece_id=XFLQI3GPFZHLMLEFBIMXMA2SC6P7VWVNIKYEWLHOKDUZQW5KFHMA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:10.290945294 (497h59m10.16202835s) [upload=437071]: Next returned nil segment Dec 26 09:28:10.290958509 (497h59m10.162041554s) [upload=437071]: Waiting for error group managing segments... Dec 26 09:28:10.678285896 (497h59m10.549368931s) [noise=true , address=145.79.227.50:4374 , piece_id=CVOZ4UIWRNBZAIN5RKEKZ32ICGZFD34TVPLDYMVJW435V3RZC6CA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.681122548 (497h59m10.552205583s) [noise=true , address=145.79.227.42:4122 , piece_id=6K4GMP2A4VAOO3Y6LPALBPXBVWLA54AR4SM5MA4YTETBBY2B752A, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.683302087 (497h59m10.554385132s) [noise=true , address=145.79.227.4:4186 , piece_id=7O3BVVNANVOCSC4OZY5Q3YMXGXYE4OUEBCPVT4PYMMXVCHPRYZHA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.683990328 (497h59m10.555073363s) [noise=true , address=145.79.227.52:4447 , piece_id=QPOPFXJLXURCKFPXWSE7HQPXEZRV3XZ5R74BP55C3J6E67U7MK6Q, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.693692112 (497h59m10.564775157s) [noise=true , address=145.79.227.39:4014 , piece_id=Y5AZV5RYKECLQWNIBC7ETIMM274GE2AV3M5I3MNJA5PTWCCHA5HA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.696002942 (497h59m10.567085987s) [noise=true , address=145.79.227.45:4231 , piece_id=WOXIAN5R7RTWOUIFD7GWRPRQHNUA6S6BUAYPH4STJ4LV2VAEGXFA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.697317195 (497h59m10.56840023s) [noise=true , address=145.79.227.11:4731 , piece_id=XJPXNJQHVIYLEPV22DZYULXUVMWHEN5WSGJLRLN6BPC2XZB4MOCQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.705818693 (497h59m10.576901738s) [noise=true , address=145.79.227.38:4979 , piece_id=BVH2BXNSN2G3MNESWFVFKI52KS72DDB6TWTJPDTFNIDODIEJXGSA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.713470103 (497h59m10.584553148s) [noise=true , address=145.79.227.46:4603 , piece_id=YFLTIC3TDVZL5VSEVPTZA4OJCPNODT3ZQ3QVLEK2T7GBHM7WAKTQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.955672247 (497h59m10.826755292s) [noise=true , address=45.91.116.14:4189 , piece_id=XFLQI3GPFZHLMLEFBIMXMA2SC6P7VWVNIKYEWLHOKDUZQW5KFHMA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.992053222 (497h59m10.863136257s) [noise=true , address=45.91.116.12:4226 , piece_id=O372B7PCAKW4AUAHRYBHBVGVLE2W734JCS6UNXEZJQHLC3K6DDZQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.024551975 (497h59m10.89563502s) [noise=true , address=45.91.116.8:4046 , piece_id=JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.031430964 (497h59m10.902514009s) [noise=true , address=45.91.116.10:4081 , piece_id=DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.031476941 (497h59m10.902559987s) [seg_pos={0 0}, upload=437071]: Segment reached optimal threshold of 13 pieces. Dec 26 09:28:11.031753057 (497h59m10.902836102s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection Dec 26 09:28:11.031803594 (497h59m10.902886659s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032208154 (497h59m10.903291189s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 09:28:11.032217091 (497h59m10.903300126s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032346407 (497h59m10.903429452s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection Dec 26 09:28:11.032354913 (497h59m10.903437958s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032372537 (497h59m10.903455582s) [seg_pos={0 0}, upload=437071]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:28:11.032470764 (497h59m10.903553799s) [upload=437071]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:28:11.032477387 (497h59m10.903560432s) [upload=437071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:28:11.319582514 (497h59m11.190665559s) [upload=437071]: Done uploading segments... Dec 26 09:28:11.319601119 (497h59m11.190684154s) [upload=437071]: Upload finished. err: <nil> Duration: 5.726579986s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [................X....XXX..X.......X.....XX.XX.XXX.....X..X..] duration: 6.063437846s trace: https://jaeger.dev.storj.io/trace/319e22bd076acae8 FAIL (2025-12-26 09:30:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ) context deadline exceeded ======LOGS====== oad=437071]: Uploading segments... Dec 26 09:28:08.418955570 (497h59m8.290038605s) [upload=437071]: Got next segment. Inline: false Dec 26 09:28:08.419031294 (497h59m8.290114329s) [upload=437071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:28:08.419036804 (497h59m8.290119839s) [upload=437071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:28:08.663968175 (497h59m8.53505126s) [seg_pos={0 0}, upload=437071]: Begin upload segment... Dec 26 09:28:08.664026606 (497h59m8.535109642s) [seg_pos={0 0}, upload=437071]: Done begin upload segment. Dec 26 09:28:08.664068326 (497h59m8.535151371s) [noise=true , address=145.79.227.52:4447 , piece_id=QPOPFXJLXURCKFPXWSE7HQPXEZRV3XZ5R74BP55C3J6E67U7MK6Q, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.664145823 (497h59m8.535228868s) [noise=true , address=145.79.227.11:4731 , piece_id=XJPXNJQHVIYLEPV22DZYULXUVMWHEN5WSGJLRLN6BPC2XZB4MOCQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.664921371 (497h59m8.536004416s) [noise=true , address=145.79.227.4:4186 , piece_id=7O3BVVNANVOCSC4OZY5Q3YMXGXYE4OUEBCPVT4PYMMXVCHPRYZHA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665214528 (497h59m8.536297573s) [noise=true , address=45.91.116.10:4081 , piece_id=DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665614420 (497h59m8.536697455s) [noise=true , address=45.91.116.8:4046 , piece_id=JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665993302 (497h59m8.537076337s) [noise=true , address=45.91.116.12:4226 , piece_id=O372B7PCAKW4AUAHRYBHBVGVLE2W734JCS6UNXEZJQHLC3K6DDZQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666040852 (497h59m8.537123887s) [noise=true , address=145.79.227.42:4122 , piece_id=6K4GMP2A4VAOO3Y6LPALBPXBVWLA54AR4SM5MA4YTETBBY2B752A, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666405606 (497h59m8.537488651s) [noise=true , address=145.79.227.50:4374 , piece_id=CVOZ4UIWRNBZAIN5RKEKZ32ICGZFD34TVPLDYMVJW435V3RZC6CA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666702762 (497h59m8.537785798s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666995319 (497h59m8.538078364s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667420760 (497h59m8.538503795s) [noise=true , address=145.79.227.38:4979 , piece_id=BVH2BXNSN2G3MNESWFVFKI52KS72DDB6TWTJPDTFNIDODIEJXGSA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667791636 (497h59m8.538874671s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667930320 (497h59m8.539013365s) [noise=true , address=145.79.227.46:4603 , piece_id=YFLTIC3TDVZL5VSEVPTZA4OJCPNODT3ZQ3QVLEK2T7GBHM7WAKTQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.668574898 (497h59m8.539657933s) [noise=true , address=145.79.227.45:4231 , piece_id=WOXIAN5R7RTWOUIFD7GWRPRQHNUA6S6BUAYPH4STJ4LV2VAEGXFA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.668945613 (497h59m8.540028658s) [noise=true , address=145.79.227.39:4014 , piece_id=Y5AZV5RYKECLQWNIBC7ETIMM274GE2AV3M5I3MNJA5PTWCCHA5HA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.669338902 (497h59m8.540421937s) [noise=true , address=45.91.116.14:4189 , piece_id=XFLQI3GPFZHLMLEFBIMXMA2SC6P7VWVNIKYEWLHOKDUZQW5KFHMA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:10.290945294 (497h59m10.16202835s) [upload=437071]: Next returned nil segment Dec 26 09:28:10.290958509 (497h59m10.162041554s) [upload=437071]: Waiting for error group managing segments... Dec 26 09:28:10.678285896 (497h59m10.549368931s) [noise=true , address=145.79.227.50:4374 , piece_id=CVOZ4UIWRNBZAIN5RKEKZ32ICGZFD34TVPLDYMVJW435V3RZC6CA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.681122548 (497h59m10.552205583s) [noise=true , address=145.79.227.42:4122 , piece_id=6K4GMP2A4VAOO3Y6LPALBPXBVWLA54AR4SM5MA4YTETBBY2B752A, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.683302087 (497h59m10.554385132s) [noise=true , address=145.79.227.4:4186 , piece_id=7O3BVVNANVOCSC4OZY5Q3YMXGXYE4OUEBCPVT4PYMMXVCHPRYZHA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.683990328 (497h59m10.555073363s) [noise=true , address=145.79.227.52:4447 , piece_id=QPOPFXJLXURCKFPXWSE7HQPXEZRV3XZ5R74BP55C3J6E67U7MK6Q, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.693692112 (497h59m10.564775157s) [noise=true , address=145.79.227.39:4014 , piece_id=Y5AZV5RYKECLQWNIBC7ETIMM274GE2AV3M5I3MNJA5PTWCCHA5HA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.696002942 (497h59m10.567085987s) [noise=true , address=145.79.227.45:4231 , piece_id=WOXIAN5R7RTWOUIFD7GWRPRQHNUA6S6BUAYPH4STJ4LV2VAEGXFA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.697317195 (497h59m10.56840023s) [noise=true , address=145.79.227.11:4731 , piece_id=XJPXNJQHVIYLEPV22DZYULXUVMWHEN5WSGJLRLN6BPC2XZB4MOCQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.705818693 (497h59m10.576901738s) [noise=true , address=145.79.227.38:4979 , piece_id=BVH2BXNSN2G3MNESWFVFKI52KS72DDB6TWTJPDTFNIDODIEJXGSA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.713470103 (497h59m10.584553148s) [noise=true , address=145.79.227.46:4603 , piece_id=YFLTIC3TDVZL5VSEVPTZA4OJCPNODT3ZQ3QVLEK2T7GBHM7WAKTQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.955672247 (497h59m10.826755292s) [noise=true , address=45.91.116.14:4189 , piece_id=XFLQI3GPFZHLMLEFBIMXMA2SC6P7VWVNIKYEWLHOKDUZQW5KFHMA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.992053222 (497h59m10.863136257s) [noise=true , address=45.91.116.12:4226 , piece_id=O372B7PCAKW4AUAHRYBHBVGVLE2W734JCS6UNXEZJQHLC3K6DDZQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.024551975 (497h59m10.89563502s) [noise=true , address=45.91.116.8:4046 , piece_id=JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.031430964 (497h59m10.902514009s) [noise=true , address=45.91.116.10:4081 , piece_id=DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.031476941 (497h59m10.902559987s) [seg_pos={0 0}, upload=437071]: Segment reached optimal threshold of 13 pieces. Dec 26 09:28:11.031753057 (497h59m10.902836102s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection Dec 26 09:28:11.031803594 (497h59m10.902886659s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032208154 (497h59m10.903291189s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 09:28:11.032217091 (497h59m10.903300126s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032346407 (497h59m10.903429452s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection Dec 26 09:28:11.032354913 (497h59m10.903437958s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032372537 (497h59m10.903455582s) [seg_pos={0 0}, upload=437071]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:28:11.032470764 (497h59m10.903553799s) [upload=437071]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:28:11.032477387 (497h59m10.903560432s) [upload=437071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:28:11.319582514 (497h59m11.190665559s) [upload=437071]: Done uploading segments... Dec 26 09:28:11.319601119 (497h59m11.190684154s) [upload=437071]: Upload finished. err: <nil> Duration: 6.063472714s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.................X....XXX..X.......X.....XX.XX.XXX.....X..X.] duration: 5.756415175s trace: https://jaeger.dev.storj.io/trace/319e22bd076acae8 FAIL (2025-12-26 09:29:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ) context deadline exceeded ======LOGS====== oad=437071]: Uploading segments... Dec 26 09:28:08.418955570 (497h59m8.290038605s) [upload=437071]: Got next segment. Inline: false Dec 26 09:28:08.419031294 (497h59m8.290114329s) [upload=437071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:28:08.419036804 (497h59m8.290119839s) [upload=437071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:28:08.663968175 (497h59m8.53505126s) [seg_pos={0 0}, upload=437071]: Begin upload segment... Dec 26 09:28:08.664026606 (497h59m8.535109642s) [seg_pos={0 0}, upload=437071]: Done begin upload segment. Dec 26 09:28:08.664068326 (497h59m8.535151371s) [noise=true , address=145.79.227.52:4447 , piece_id=QPOPFXJLXURCKFPXWSE7HQPXEZRV3XZ5R74BP55C3J6E67U7MK6Q, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.664145823 (497h59m8.535228868s) [noise=true , address=145.79.227.11:4731 , piece_id=XJPXNJQHVIYLEPV22DZYULXUVMWHEN5WSGJLRLN6BPC2XZB4MOCQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.664921371 (497h59m8.536004416s) [noise=true , address=145.79.227.4:4186 , piece_id=7O3BVVNANVOCSC4OZY5Q3YMXGXYE4OUEBCPVT4PYMMXVCHPRYZHA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665214528 (497h59m8.536297573s) [noise=true , address=45.91.116.10:4081 , piece_id=DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665614420 (497h59m8.536697455s) [noise=true , address=45.91.116.8:4046 , piece_id=JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665993302 (497h59m8.537076337s) [noise=true , address=45.91.116.12:4226 , piece_id=O372B7PCAKW4AUAHRYBHBVGVLE2W734JCS6UNXEZJQHLC3K6DDZQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666040852 (497h59m8.537123887s) [noise=true , address=145.79.227.42:4122 , piece_id=6K4GMP2A4VAOO3Y6LPALBPXBVWLA54AR4SM5MA4YTETBBY2B752A, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666405606 (497h59m8.537488651s) [noise=true , address=145.79.227.50:4374 , piece_id=CVOZ4UIWRNBZAIN5RKEKZ32ICGZFD34TVPLDYMVJW435V3RZC6CA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666702762 (497h59m8.537785798s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666995319 (497h59m8.538078364s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667420760 (497h59m8.538503795s) [noise=true , address=145.79.227.38:4979 , piece_id=BVH2BXNSN2G3MNESWFVFKI52KS72DDB6TWTJPDTFNIDODIEJXGSA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667791636 (497h59m8.538874671s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667930320 (497h59m8.539013365s) [noise=true , address=145.79.227.46:4603 , piece_id=YFLTIC3TDVZL5VSEVPTZA4OJCPNODT3ZQ3QVLEK2T7GBHM7WAKTQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.668574898 (497h59m8.539657933s) [noise=true , address=145.79.227.45:4231 , piece_id=WOXIAN5R7RTWOUIFD7GWRPRQHNUA6S6BUAYPH4STJ4LV2VAEGXFA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.668945613 (497h59m8.540028658s) [noise=true , address=145.79.227.39:4014 , piece_id=Y5AZV5RYKECLQWNIBC7ETIMM274GE2AV3M5I3MNJA5PTWCCHA5HA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.669338902 (497h59m8.540421937s) [noise=true , address=45.91.116.14:4189 , piece_id=XFLQI3GPFZHLMLEFBIMXMA2SC6P7VWVNIKYEWLHOKDUZQW5KFHMA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:10.290945294 (497h59m10.16202835s) [upload=437071]: Next returned nil segment Dec 26 09:28:10.290958509 (497h59m10.162041554s) [upload=437071]: Waiting for error group managing segments... Dec 26 09:28:10.678285896 (497h59m10.549368931s) [noise=true , address=145.79.227.50:4374 , piece_id=CVOZ4UIWRNBZAIN5RKEKZ32ICGZFD34TVPLDYMVJW435V3RZC6CA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.681122548 (497h59m10.552205583s) [noise=true , address=145.79.227.42:4122 , piece_id=6K4GMP2A4VAOO3Y6LPALBPXBVWLA54AR4SM5MA4YTETBBY2B752A, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.683302087 (497h59m10.554385132s) [noise=true , address=145.79.227.4:4186 , piece_id=7O3BVVNANVOCSC4OZY5Q3YMXGXYE4OUEBCPVT4PYMMXVCHPRYZHA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.683990328 (497h59m10.555073363s) [noise=true , address=145.79.227.52:4447 , piece_id=QPOPFXJLXURCKFPXWSE7HQPXEZRV3XZ5R74BP55C3J6E67U7MK6Q, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.693692112 (497h59m10.564775157s) [noise=true , address=145.79.227.39:4014 , piece_id=Y5AZV5RYKECLQWNIBC7ETIMM274GE2AV3M5I3MNJA5PTWCCHA5HA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.696002942 (497h59m10.567085987s) [noise=true , address=145.79.227.45:4231 , piece_id=WOXIAN5R7RTWOUIFD7GWRPRQHNUA6S6BUAYPH4STJ4LV2VAEGXFA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.697317195 (497h59m10.56840023s) [noise=true , address=145.79.227.11:4731 , piece_id=XJPXNJQHVIYLEPV22DZYULXUVMWHEN5WSGJLRLN6BPC2XZB4MOCQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.705818693 (497h59m10.576901738s) [noise=true , address=145.79.227.38:4979 , piece_id=BVH2BXNSN2G3MNESWFVFKI52KS72DDB6TWTJPDTFNIDODIEJXGSA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.713470103 (497h59m10.584553148s) [noise=true , address=145.79.227.46:4603 , piece_id=YFLTIC3TDVZL5VSEVPTZA4OJCPNODT3ZQ3QVLEK2T7GBHM7WAKTQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.955672247 (497h59m10.826755292s) [noise=true , address=45.91.116.14:4189 , piece_id=XFLQI3GPFZHLMLEFBIMXMA2SC6P7VWVNIKYEWLHOKDUZQW5KFHMA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.992053222 (497h59m10.863136257s) [noise=true , address=45.91.116.12:4226 , piece_id=O372B7PCAKW4AUAHRYBHBVGVLE2W734JCS6UNXEZJQHLC3K6DDZQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.024551975 (497h59m10.89563502s) [noise=true , address=45.91.116.8:4046 , piece_id=JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.031430964 (497h59m10.902514009s) [noise=true , address=45.91.116.10:4081 , piece_id=DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.031476941 (497h59m10.902559987s) [seg_pos={0 0}, upload=437071]: Segment reached optimal threshold of 13 pieces. Dec 26 09:28:11.031753057 (497h59m10.902836102s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection Dec 26 09:28:11.031803594 (497h59m10.902886659s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032208154 (497h59m10.903291189s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 09:28:11.032217091 (497h59m10.903300126s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032346407 (497h59m10.903429452s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection Dec 26 09:28:11.032354913 (497h59m10.903437958s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032372537 (497h59m10.903455582s) [seg_pos={0 0}, upload=437071]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:28:11.032470764 (497h59m10.903553799s) [upload=437071]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:28:11.032477387 (497h59m10.903560432s) [upload=437071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:28:11.319582514 (497h59m11.190665559s) [upload=437071]: Done uploading segments... Dec 26 09:28:11.319601119 (497h59m11.190684154s) [upload=437071]: Upload finished. err: <nil> Duration: 5.75661665s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..................X....XXX..X.......X.....XX.XX.XXX.....X..X] duration: 30.002856251s trace: https://jaeger.dev.storj.io/trace/319e22bd076acae8 FAIL (2025-12-26 09:28:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ) context deadline exceeded ======LOGS====== oad=437071]: Uploading segments... Dec 26 09:28:08.418955570 (497h59m8.290038605s) [upload=437071]: Got next segment. Inline: false Dec 26 09:28:08.419031294 (497h59m8.290114329s) [upload=437071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:28:08.419036804 (497h59m8.290119839s) [upload=437071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:28:08.663968175 (497h59m8.53505126s) [seg_pos={0 0}, upload=437071]: Begin upload segment... Dec 26 09:28:08.664026606 (497h59m8.535109642s) [seg_pos={0 0}, upload=437071]: Done begin upload segment. Dec 26 09:28:08.664068326 (497h59m8.535151371s) [noise=true , address=145.79.227.52:4447 , piece_id=QPOPFXJLXURCKFPXWSE7HQPXEZRV3XZ5R74BP55C3J6E67U7MK6Q, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.664145823 (497h59m8.535228868s) [noise=true , address=145.79.227.11:4731 , piece_id=XJPXNJQHVIYLEPV22DZYULXUVMWHEN5WSGJLRLN6BPC2XZB4MOCQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.664921371 (497h59m8.536004416s) [noise=true , address=145.79.227.4:4186 , piece_id=7O3BVVNANVOCSC4OZY5Q3YMXGXYE4OUEBCPVT4PYMMXVCHPRYZHA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665214528 (497h59m8.536297573s) [noise=true , address=45.91.116.10:4081 , piece_id=DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665614420 (497h59m8.536697455s) [noise=true , address=45.91.116.8:4046 , piece_id=JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665993302 (497h59m8.537076337s) [noise=true , address=45.91.116.12:4226 , piece_id=O372B7PCAKW4AUAHRYBHBVGVLE2W734JCS6UNXEZJQHLC3K6DDZQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666040852 (497h59m8.537123887s) [noise=true , address=145.79.227.42:4122 , piece_id=6K4GMP2A4VAOO3Y6LPALBPXBVWLA54AR4SM5MA4YTETBBY2B752A, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666405606 (497h59m8.537488651s) [noise=true , address=145.79.227.50:4374 , piece_id=CVOZ4UIWRNBZAIN5RKEKZ32ICGZFD34TVPLDYMVJW435V3RZC6CA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666702762 (497h59m8.537785798s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666995319 (497h59m8.538078364s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667420760 (497h59m8.538503795s) [noise=true , address=145.79.227.38:4979 , piece_id=BVH2BXNSN2G3MNESWFVFKI52KS72DDB6TWTJPDTFNIDODIEJXGSA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667791636 (497h59m8.538874671s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667930320 (497h59m8.539013365s) [noise=true , address=145.79.227.46:4603 , piece_id=YFLTIC3TDVZL5VSEVPTZA4OJCPNODT3ZQ3QVLEK2T7GBHM7WAKTQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.668574898 (497h59m8.539657933s) [noise=true , address=145.79.227.45:4231 , piece_id=WOXIAN5R7RTWOUIFD7GWRPRQHNUA6S6BUAYPH4STJ4LV2VAEGXFA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.668945613 (497h59m8.540028658s) [noise=true , address=145.79.227.39:4014 , piece_id=Y5AZV5RYKECLQWNIBC7ETIMM274GE2AV3M5I3MNJA5PTWCCHA5HA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.669338902 (497h59m8.540421937s) [noise=true , address=45.91.116.14:4189 , piece_id=XFLQI3GPFZHLMLEFBIMXMA2SC6P7VWVNIKYEWLHOKDUZQW5KFHMA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:10.290945294 (497h59m10.16202835s) [upload=437071]: Next returned nil segment Dec 26 09:28:10.290958509 (497h59m10.162041554s) [upload=437071]: Waiting for error group managing segments... Dec 26 09:28:10.678285896 (497h59m10.549368931s) [noise=true , address=145.79.227.50:4374 , piece_id=CVOZ4UIWRNBZAIN5RKEKZ32ICGZFD34TVPLDYMVJW435V3RZC6CA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.681122548 (497h59m10.552205583s) [noise=true , address=145.79.227.42:4122 , piece_id=6K4GMP2A4VAOO3Y6LPALBPXBVWLA54AR4SM5MA4YTETBBY2B752A, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.683302087 (497h59m10.554385132s) [noise=true , address=145.79.227.4:4186 , piece_id=7O3BVVNANVOCSC4OZY5Q3YMXGXYE4OUEBCPVT4PYMMXVCHPRYZHA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.683990328 (497h59m10.555073363s) [noise=true , address=145.79.227.52:4447 , piece_id=QPOPFXJLXURCKFPXWSE7HQPXEZRV3XZ5R74BP55C3J6E67U7MK6Q, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.693692112 (497h59m10.564775157s) [noise=true , address=145.79.227.39:4014 , piece_id=Y5AZV5RYKECLQWNIBC7ETIMM274GE2AV3M5I3MNJA5PTWCCHA5HA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.696002942 (497h59m10.567085987s) [noise=true , address=145.79.227.45:4231 , piece_id=WOXIAN5R7RTWOUIFD7GWRPRQHNUA6S6BUAYPH4STJ4LV2VAEGXFA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.697317195 (497h59m10.56840023s) [noise=true , address=145.79.227.11:4731 , piece_id=XJPXNJQHVIYLEPV22DZYULXUVMWHEN5WSGJLRLN6BPC2XZB4MOCQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.705818693 (497h59m10.576901738s) [noise=true , address=145.79.227.38:4979 , piece_id=BVH2BXNSN2G3MNESWFVFKI52KS72DDB6TWTJPDTFNIDODIEJXGSA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.713470103 (497h59m10.584553148s) [noise=true , address=145.79.227.46:4603 , piece_id=YFLTIC3TDVZL5VSEVPTZA4OJCPNODT3ZQ3QVLEK2T7GBHM7WAKTQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.955672247 (497h59m10.826755292s) [noise=true , address=45.91.116.14:4189 , piece_id=XFLQI3GPFZHLMLEFBIMXMA2SC6P7VWVNIKYEWLHOKDUZQW5KFHMA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.992053222 (497h59m10.863136257s) [noise=true , address=45.91.116.12:4226 , piece_id=O372B7PCAKW4AUAHRYBHBVGVLE2W734JCS6UNXEZJQHLC3K6DDZQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.024551975 (497h59m10.89563502s) [noise=true , address=45.91.116.8:4046 , piece_id=JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.031430964 (497h59m10.902514009s) [noise=true , address=45.91.116.10:4081 , piece_id=DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.031476941 (497h59m10.902559987s) [seg_pos={0 0}, upload=437071]: Segment reached optimal threshold of 13 pieces. Dec 26 09:28:11.031753057 (497h59m10.902836102s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection Dec 26 09:28:11.031803594 (497h59m10.902886659s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032208154 (497h59m10.903291189s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 09:28:11.032217091 (497h59m10.903300126s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032346407 (497h59m10.903429452s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection Dec 26 09:28:11.032354913 (497h59m10.903437958s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032372537 (497h59m10.903455582s) [seg_pos={0 0}, upload=437071]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:28:11.032470764 (497h59m10.903553799s) [upload=437071]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:28:11.032477387 (497h59m10.903560432s) [upload=437071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:28:11.319582514 (497h59m11.190665559s) [upload=437071]: Done uploading segments... Dec 26 09:28:11.319601119 (497h59m11.190684154s) [upload=437071]: Upload finished. err: <nil> Duration: 30.002883935s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...................X....XXX..X.......X.....XX.XX.XXX.....X..] duration: 6.062604128s trace: https://jaeger.dev.storj.io/trace/5a6672a524f8bd5c FAIL (2025-12-26 09:27:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 26 09:25:49.895347339 (497h56m49.766430374s) [upload=437042]: Got next segment. Inline: false Dec 26 09:25:49.895382436 (497h56m49.766465471s) [upload=437042]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:25:49.895388638 (497h56m49.766471673s) [upload=437042]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:25:50.140874455 (497h56m50.0119575s) [seg_pos={0 0}, upload=437042]: Begin upload segment... Dec 26 09:25:50.140938667 (497h56m50.012021703s) [seg_pos={0 0}, upload=437042]: Done begin upload segment. Dec 26 09:25:50.140969937 (497h56m50.012052982s) [noise=true , address=45.91.116.5:4297 , piece_id=BATJXBJ3LBJUNYL6KLTI6LJA3XLIV6CGLUDNKXYMQUOO55X3SX5A, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141055701 (497h56m50.012138736s) [noise=true , address=145.79.227.12:4767 , piece_id=M4PDDCTAWGEB3QVC4DAS2ARX4KEE2LF4CKX2WSXRHQDYHTME4WZQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141560251 (497h56m50.012643286s) [noise=true , address=145.79.227.45:4229 , piece_id=CL357SM3G2AA4OL5L2OLYGPFJS7DIAPKHYL22GXN53SEORT5N64A, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141987484 (497h56m50.01307054s) [noise=true , address=145.79.227.29:4942 , piece_id=KJKCQQLMXX7Q4YTAQ3LT2QKWTD34O7VGIX5Z7XQLVMATQWQY5MTQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142370263 (497h56m50.013453298s) [noise=true , address=145.79.227.35:4482 , piece_id=RY6VOEVKAAN42A5GKDPH57Y4ZFRRHWZWJONB3ZLMQX7ZELHGXAVQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142568982 (497h56m50.013652017s) [noise=true , address=145.79.227.5:4222 , piece_id=E5E25UYXESEQSVMSHNH4XDFTH56IQZLBINAT45U5TCEO6LQ2Q5ZA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142667370 (497h56m50.013750405s) [noise=true , address=145.79.227.49:4338 , piece_id=AWMUDKGCIUXBHU7MADZ4NPX3I6IFRROMW5VWPZX7R3XQQVRSBYVQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142934909 (497h56m50.014017954s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143000143 (497h56m50.014083188s) [noise=true , address=45.91.116.11:4117 , piece_id=IJF4JGZSJ3W5AM6LG4HOH323MKXCO4FTBUOXRE4IU4L6DMNEMOPQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143229550 (497h56m50.014312585s) [noise=true , address=145.79.227.51:4410 , piece_id=IYYH3QBMSPGBAFYZSTSO67EVAPHFQUZC7LQHPSYNY5GMP2XIJ7YQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143514592 (497h56m50.014597638s) [noise=true , address=45.91.116.12:4226 , piece_id=K5DEANUOCTI56TR5C7LGRDAOEXTBCKGAYR3DY3LRJQGRBVLYOJRA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143525974 (497h56m50.014609009s) [noise=true , address=145.79.227.41:4087 , piece_id=TTSD747HZZBIQ4N75ANKAIBCUA3BLKNOOKKRG5EQJMSEQDCIBAZQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143931587 (497h56m50.015014622s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.144360102 (497h56m50.015443137s) [noise=true , address=145.79.227.36:4518 , piece_id=N5FUZ6ZZTDG5RVDLLUN4IL7M3SCOKBYSYUCTML4IT5JR2MWXAKBA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.144691474 (497h56m50.015774509s) [noise=true , address=145.79.227.52:4445 , piece_id=YUZW73ADUNPWDG2H2NPGUOP34TCV7WRR53X7LEGZMQFCEGKMSTUA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.146265421 (497h56m50.017348466s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:51.856038844 (497h56m51.727121879s) [upload=437042]: Next returned nil segment Dec 26 09:25:51.856051348 (497h56m51.727134384s) [upload=437042]: Waiting for error group managing segments... Dec 26 09:25:52.140557808 (497h56m52.011640843s) [noise=true , address=145.79.227.51:4410 , piece_id=IYYH3QBMSPGBAFYZSTSO67EVAPHFQUZC7LQHPSYNY5GMP2XIJ7YQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.153967352 (497h56m52.025050387s) [noise=true , address=145.79.227.41:4087 , piece_id=TTSD747HZZBIQ4N75ANKAIBCUA3BLKNOOKKRG5EQJMSEQDCIBAZQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.158364354 (497h56m52.029447399s) [noise=true , address=145.79.227.45:4229 , piece_id=CL357SM3G2AA4OL5L2OLYGPFJS7DIAPKHYL22GXN53SEORT5N64A, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.162378237 (497h56m52.033461283s) [noise=true , address=145.79.227.49:4338 , piece_id=AWMUDKGCIUXBHU7MADZ4NPX3I6IFRROMW5VWPZX7R3XQQVRSBYVQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.164871355 (497h56m52.03595439s) [noise=true , address=145.79.227.12:4767 , piece_id=M4PDDCTAWGEB3QVC4DAS2ARX4KEE2LF4CKX2WSXRHQDYHTME4WZQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.171507481 (497h56m52.042590526s) [noise=true , address=145.79.227.52:4445 , piece_id=YUZW73ADUNPWDG2H2NPGUOP34TCV7WRR53X7LEGZMQFCEGKMSTUA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.172026720 (497h56m52.043109755s) [noise=true , address=145.79.227.35:4482 , piece_id=RY6VOEVKAAN42A5GKDPH57Y4ZFRRHWZWJONB3ZLMQX7ZELHGXAVQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.174049311 (497h56m52.045132356s) [noise=true , address=145.79.227.36:4518 , piece_id=N5FUZ6ZZTDG5RVDLLUN4IL7M3SCOKBYSYUCTML4IT5JR2MWXAKBA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.178618181 (497h56m52.049701226s) [noise=true , address=145.79.227.29:4942 , piece_id=KJKCQQLMXX7Q4YTAQ3LT2QKWTD34O7VGIX5Z7XQLVMATQWQY5MTQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.190615577 (497h56m52.061698612s) [noise=true , address=145.79.227.5:4222 , piece_id=E5E25UYXESEQSVMSHNH4XDFTH56IQZLBINAT45U5TCEO6LQ2Q5ZA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.399223531 (497h56m52.270306567s) [noise=true , address=45.91.116.12:4226 , piece_id=K5DEANUOCTI56TR5C7LGRDAOEXTBCKGAYR3DY3LRJQGRBVLYOJRA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.460637806 (497h56m52.331720841s) [noise=true , address=45.91.116.11:4117 , piece_id=IJF4JGZSJ3W5AM6LG4HOH323MKXCO4FTBUOXRE4IU4L6DMNEMOPQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.482636275 (497h56m52.35371931s) [noise=true , address=45.91.116.5:4297 , piece_id=BATJXBJ3LBJUNYL6KLTI6LJA3XLIV6CGLUDNKXYMQUOO55X3SX5A, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.482655711 (497h56m52.353738746s) [seg_pos={0 0}, upload=437042]: Segment reached optimal threshold of 13 pieces. Dec 26 09:25:52.482830083 (497h56m52.353913128s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 09:25:52.482836605 (497h56m52.35391964s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483038169 (497h56m52.354121204s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection Dec 26 09:25:52.483044140 (497h56m52.354127176s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483447479 (497h56m52.354530524s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection Dec 26 09:25:52.483468148 (497h56m52.354551203s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483549814 (497h56m52.354632869s) [seg_pos={0 0}, upload=437042]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:25:52.483819818 (497h56m52.354902863s) [upload=437042]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:25:52.483828595 (497h56m52.35491164s) [upload=437042]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:25:52.771102113 (497h56m52.642185158s) [upload=437042]: Done uploading segments... Dec 26 09:25:52.771131258 (497h56m52.642214303s) [upload=437042]: Upload finished. err: <nil> Duration: 6.06263213s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....................X....XXX..X.......X.....XX.XX.XXX.....X.] duration: 5.746081271s trace: https://jaeger.dev.storj.io/trace/5a6672a524f8bd5c FAIL (2025-12-26 09:26:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 26 09:25:49.895347339 (497h56m49.766430374s) [upload=437042]: Got next segment. Inline: false Dec 26 09:25:49.895382436 (497h56m49.766465471s) [upload=437042]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:25:49.895388638 (497h56m49.766471673s) [upload=437042]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:25:50.140874455 (497h56m50.0119575s) [seg_pos={0 0}, upload=437042]: Begin upload segment... Dec 26 09:25:50.140938667 (497h56m50.012021703s) [seg_pos={0 0}, upload=437042]: Done begin upload segment. Dec 26 09:25:50.140969937 (497h56m50.012052982s) [noise=true , address=45.91.116.5:4297 , piece_id=BATJXBJ3LBJUNYL6KLTI6LJA3XLIV6CGLUDNKXYMQUOO55X3SX5A, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141055701 (497h56m50.012138736s) [noise=true , address=145.79.227.12:4767 , piece_id=M4PDDCTAWGEB3QVC4DAS2ARX4KEE2LF4CKX2WSXRHQDYHTME4WZQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141560251 (497h56m50.012643286s) [noise=true , address=145.79.227.45:4229 , piece_id=CL357SM3G2AA4OL5L2OLYGPFJS7DIAPKHYL22GXN53SEORT5N64A, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141987484 (497h56m50.01307054s) [noise=true , address=145.79.227.29:4942 , piece_id=KJKCQQLMXX7Q4YTAQ3LT2QKWTD34O7VGIX5Z7XQLVMATQWQY5MTQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142370263 (497h56m50.013453298s) [noise=true , address=145.79.227.35:4482 , piece_id=RY6VOEVKAAN42A5GKDPH57Y4ZFRRHWZWJONB3ZLMQX7ZELHGXAVQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142568982 (497h56m50.013652017s) [noise=true , address=145.79.227.5:4222 , piece_id=E5E25UYXESEQSVMSHNH4XDFTH56IQZLBINAT45U5TCEO6LQ2Q5ZA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142667370 (497h56m50.013750405s) [noise=true , address=145.79.227.49:4338 , piece_id=AWMUDKGCIUXBHU7MADZ4NPX3I6IFRROMW5VWPZX7R3XQQVRSBYVQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142934909 (497h56m50.014017954s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143000143 (497h56m50.014083188s) [noise=true , address=45.91.116.11:4117 , piece_id=IJF4JGZSJ3W5AM6LG4HOH323MKXCO4FTBUOXRE4IU4L6DMNEMOPQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143229550 (497h56m50.014312585s) [noise=true , address=145.79.227.51:4410 , piece_id=IYYH3QBMSPGBAFYZSTSO67EVAPHFQUZC7LQHPSYNY5GMP2XIJ7YQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143514592 (497h56m50.014597638s) [noise=true , address=45.91.116.12:4226 , piece_id=K5DEANUOCTI56TR5C7LGRDAOEXTBCKGAYR3DY3LRJQGRBVLYOJRA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143525974 (497h56m50.014609009s) [noise=true , address=145.79.227.41:4087 , piece_id=TTSD747HZZBIQ4N75ANKAIBCUA3BLKNOOKKRG5EQJMSEQDCIBAZQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143931587 (497h56m50.015014622s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.144360102 (497h56m50.015443137s) [noise=true , address=145.79.227.36:4518 , piece_id=N5FUZ6ZZTDG5RVDLLUN4IL7M3SCOKBYSYUCTML4IT5JR2MWXAKBA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.144691474 (497h56m50.015774509s) [noise=true , address=145.79.227.52:4445 , piece_id=YUZW73ADUNPWDG2H2NPGUOP34TCV7WRR53X7LEGZMQFCEGKMSTUA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.146265421 (497h56m50.017348466s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:51.856038844 (497h56m51.727121879s) [upload=437042]: Next returned nil segment Dec 26 09:25:51.856051348 (497h56m51.727134384s) [upload=437042]: Waiting for error group managing segments... Dec 26 09:25:52.140557808 (497h56m52.011640843s) [noise=true , address=145.79.227.51:4410 , piece_id=IYYH3QBMSPGBAFYZSTSO67EVAPHFQUZC7LQHPSYNY5GMP2XIJ7YQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.153967352 (497h56m52.025050387s) [noise=true , address=145.79.227.41:4087 , piece_id=TTSD747HZZBIQ4N75ANKAIBCUA3BLKNOOKKRG5EQJMSEQDCIBAZQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.158364354 (497h56m52.029447399s) [noise=true , address=145.79.227.45:4229 , piece_id=CL357SM3G2AA4OL5L2OLYGPFJS7DIAPKHYL22GXN53SEORT5N64A, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.162378237 (497h56m52.033461283s) [noise=true , address=145.79.227.49:4338 , piece_id=AWMUDKGCIUXBHU7MADZ4NPX3I6IFRROMW5VWPZX7R3XQQVRSBYVQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.164871355 (497h56m52.03595439s) [noise=true , address=145.79.227.12:4767 , piece_id=M4PDDCTAWGEB3QVC4DAS2ARX4KEE2LF4CKX2WSXRHQDYHTME4WZQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.171507481 (497h56m52.042590526s) [noise=true , address=145.79.227.52:4445 , piece_id=YUZW73ADUNPWDG2H2NPGUOP34TCV7WRR53X7LEGZMQFCEGKMSTUA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.172026720 (497h56m52.043109755s) [noise=true , address=145.79.227.35:4482 , piece_id=RY6VOEVKAAN42A5GKDPH57Y4ZFRRHWZWJONB3ZLMQX7ZELHGXAVQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.174049311 (497h56m52.045132356s) [noise=true , address=145.79.227.36:4518 , piece_id=N5FUZ6ZZTDG5RVDLLUN4IL7M3SCOKBYSYUCTML4IT5JR2MWXAKBA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.178618181 (497h56m52.049701226s) [noise=true , address=145.79.227.29:4942 , piece_id=KJKCQQLMXX7Q4YTAQ3LT2QKWTD34O7VGIX5Z7XQLVMATQWQY5MTQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.190615577 (497h56m52.061698612s) [noise=true , address=145.79.227.5:4222 , piece_id=E5E25UYXESEQSVMSHNH4XDFTH56IQZLBINAT45U5TCEO6LQ2Q5ZA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.399223531 (497h56m52.270306567s) [noise=true , address=45.91.116.12:4226 , piece_id=K5DEANUOCTI56TR5C7LGRDAOEXTBCKGAYR3DY3LRJQGRBVLYOJRA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.460637806 (497h56m52.331720841s) [noise=true , address=45.91.116.11:4117 , piece_id=IJF4JGZSJ3W5AM6LG4HOH323MKXCO4FTBUOXRE4IU4L6DMNEMOPQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.482636275 (497h56m52.35371931s) [noise=true , address=45.91.116.5:4297 , piece_id=BATJXBJ3LBJUNYL6KLTI6LJA3XLIV6CGLUDNKXYMQUOO55X3SX5A, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.482655711 (497h56m52.353738746s) [seg_pos={0 0}, upload=437042]: Segment reached optimal threshold of 13 pieces. Dec 26 09:25:52.482830083 (497h56m52.353913128s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 09:25:52.482836605 (497h56m52.35391964s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483038169 (497h56m52.354121204s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection Dec 26 09:25:52.483044140 (497h56m52.354127176s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483447479 (497h56m52.354530524s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection Dec 26 09:25:52.483468148 (497h56m52.354551203s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483549814 (497h56m52.354632869s) [seg_pos={0 0}, upload=437042]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:25:52.483819818 (497h56m52.354902863s) [upload=437042]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:25:52.483828595 (497h56m52.35491164s) [upload=437042]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:25:52.771102113 (497h56m52.642185158s) [upload=437042]: Done uploading segments... Dec 26 09:25:52.771131258 (497h56m52.642214303s) [upload=437042]: Upload finished. err: <nil> Duration: 5.74609694s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X....................X....XXX..X.......X.....XX.XX.XXX.....X] duration: 30.003748748s trace: https://jaeger.dev.storj.io/trace/5a6672a524f8bd5c FAIL (2025-12-26 09:26:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 26 09:25:49.895347339 (497h56m49.766430374s) [upload=437042]: Got next segment. Inline: false Dec 26 09:25:49.895382436 (497h56m49.766465471s) [upload=437042]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:25:49.895388638 (497h56m49.766471673s) [upload=437042]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:25:50.140874455 (497h56m50.0119575s) [seg_pos={0 0}, upload=437042]: Begin upload segment... Dec 26 09:25:50.140938667 (497h56m50.012021703s) [seg_pos={0 0}, upload=437042]: Done begin upload segment. Dec 26 09:25:50.140969937 (497h56m50.012052982s) [noise=true , address=45.91.116.5:4297 , piece_id=BATJXBJ3LBJUNYL6KLTI6LJA3XLIV6CGLUDNKXYMQUOO55X3SX5A, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141055701 (497h56m50.012138736s) [noise=true , address=145.79.227.12:4767 , piece_id=M4PDDCTAWGEB3QVC4DAS2ARX4KEE2LF4CKX2WSXRHQDYHTME4WZQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141560251 (497h56m50.012643286s) [noise=true , address=145.79.227.45:4229 , piece_id=CL357SM3G2AA4OL5L2OLYGPFJS7DIAPKHYL22GXN53SEORT5N64A, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141987484 (497h56m50.01307054s) [noise=true , address=145.79.227.29:4942 , piece_id=KJKCQQLMXX7Q4YTAQ3LT2QKWTD34O7VGIX5Z7XQLVMATQWQY5MTQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142370263 (497h56m50.013453298s) [noise=true , address=145.79.227.35:4482 , piece_id=RY6VOEVKAAN42A5GKDPH57Y4ZFRRHWZWJONB3ZLMQX7ZELHGXAVQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142568982 (497h56m50.013652017s) [noise=true , address=145.79.227.5:4222 , piece_id=E5E25UYXESEQSVMSHNH4XDFTH56IQZLBINAT45U5TCEO6LQ2Q5ZA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142667370 (497h56m50.013750405s) [noise=true , address=145.79.227.49:4338 , piece_id=AWMUDKGCIUXBHU7MADZ4NPX3I6IFRROMW5VWPZX7R3XQQVRSBYVQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142934909 (497h56m50.014017954s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143000143 (497h56m50.014083188s) [noise=true , address=45.91.116.11:4117 , piece_id=IJF4JGZSJ3W5AM6LG4HOH323MKXCO4FTBUOXRE4IU4L6DMNEMOPQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143229550 (497h56m50.014312585s) [noise=true , address=145.79.227.51:4410 , piece_id=IYYH3QBMSPGBAFYZSTSO67EVAPHFQUZC7LQHPSYNY5GMP2XIJ7YQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143514592 (497h56m50.014597638s) [noise=true , address=45.91.116.12:4226 , piece_id=K5DEANUOCTI56TR5C7LGRDAOEXTBCKGAYR3DY3LRJQGRBVLYOJRA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143525974 (497h56m50.014609009s) [noise=true , address=145.79.227.41:4087 , piece_id=TTSD747HZZBIQ4N75ANKAIBCUA3BLKNOOKKRG5EQJMSEQDCIBAZQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143931587 (497h56m50.015014622s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.144360102 (497h56m50.015443137s) [noise=true , address=145.79.227.36:4518 , piece_id=N5FUZ6ZZTDG5RVDLLUN4IL7M3SCOKBYSYUCTML4IT5JR2MWXAKBA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.144691474 (497h56m50.015774509s) [noise=true , address=145.79.227.52:4445 , piece_id=YUZW73ADUNPWDG2H2NPGUOP34TCV7WRR53X7LEGZMQFCEGKMSTUA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.146265421 (497h56m50.017348466s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:51.856038844 (497h56m51.727121879s) [upload=437042]: Next returned nil segment Dec 26 09:25:51.856051348 (497h56m51.727134384s) [upload=437042]: Waiting for error group managing segments... Dec 26 09:25:52.140557808 (497h56m52.011640843s) [noise=true , address=145.79.227.51:4410 , piece_id=IYYH3QBMSPGBAFYZSTSO67EVAPHFQUZC7LQHPSYNY5GMP2XIJ7YQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.153967352 (497h56m52.025050387s) [noise=true , address=145.79.227.41:4087 , piece_id=TTSD747HZZBIQ4N75ANKAIBCUA3BLKNOOKKRG5EQJMSEQDCIBAZQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.158364354 (497h56m52.029447399s) [noise=true , address=145.79.227.45:4229 , piece_id=CL357SM3G2AA4OL5L2OLYGPFJS7DIAPKHYL22GXN53SEORT5N64A, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.162378237 (497h56m52.033461283s) [noise=true , address=145.79.227.49:4338 , piece_id=AWMUDKGCIUXBHU7MADZ4NPX3I6IFRROMW5VWPZX7R3XQQVRSBYVQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.164871355 (497h56m52.03595439s) [noise=true , address=145.79.227.12:4767 , piece_id=M4PDDCTAWGEB3QVC4DAS2ARX4KEE2LF4CKX2WSXRHQDYHTME4WZQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.171507481 (497h56m52.042590526s) [noise=true , address=145.79.227.52:4445 , piece_id=YUZW73ADUNPWDG2H2NPGUOP34TCV7WRR53X7LEGZMQFCEGKMSTUA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.172026720 (497h56m52.043109755s) [noise=true , address=145.79.227.35:4482 , piece_id=RY6VOEVKAAN42A5GKDPH57Y4ZFRRHWZWJONB3ZLMQX7ZELHGXAVQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.174049311 (497h56m52.045132356s) [noise=true , address=145.79.227.36:4518 , piece_id=N5FUZ6ZZTDG5RVDLLUN4IL7M3SCOKBYSYUCTML4IT5JR2MWXAKBA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.178618181 (497h56m52.049701226s) [noise=true , address=145.79.227.29:4942 , piece_id=KJKCQQLMXX7Q4YTAQ3LT2QKWTD34O7VGIX5Z7XQLVMATQWQY5MTQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.190615577 (497h56m52.061698612s) [noise=true , address=145.79.227.5:4222 , piece_id=E5E25UYXESEQSVMSHNH4XDFTH56IQZLBINAT45U5TCEO6LQ2Q5ZA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.399223531 (497h56m52.270306567s) [noise=true , address=45.91.116.12:4226 , piece_id=K5DEANUOCTI56TR5C7LGRDAOEXTBCKGAYR3DY3LRJQGRBVLYOJRA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.460637806 (497h56m52.331720841s) [noise=true , address=45.91.116.11:4117 , piece_id=IJF4JGZSJ3W5AM6LG4HOH323MKXCO4FTBUOXRE4IU4L6DMNEMOPQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.482636275 (497h56m52.35371931s) [noise=true , address=45.91.116.5:4297 , piece_id=BATJXBJ3LBJUNYL6KLTI6LJA3XLIV6CGLUDNKXYMQUOO55X3SX5A, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.482655711 (497h56m52.353738746s) [seg_pos={0 0}, upload=437042]: Segment reached optimal threshold of 13 pieces. Dec 26 09:25:52.482830083 (497h56m52.353913128s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 09:25:52.482836605 (497h56m52.35391964s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483038169 (497h56m52.354121204s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection Dec 26 09:25:52.483044140 (497h56m52.354127176s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483447479 (497h56m52.354530524s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection Dec 26 09:25:52.483468148 (497h56m52.354551203s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483549814 (497h56m52.354632869s) [seg_pos={0 0}, upload=437042]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:25:52.483819818 (497h56m52.354902863s) [upload=437042]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:25:52.483828595 (497h56m52.35491164s) [upload=437042]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:25:52.771102113 (497h56m52.642185158s) [upload=437042]: Done uploading segments... Dec 26 09:25:52.771131258 (497h56m52.642214303s) [upload=437042]: Upload finished. err: <nil> Duration: 30.003771734s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX....................X....XXX..X.......X.....XX.XX.XXX.....] duration: 5.959248379s trace: https://jaeger.dev.storj.io/trace/389ac4bc3ff91e5c FAIL (2025-12-26 09:25:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding segments... Dec 26 09:22:39.145892195 (497h53m39.01697523s) [upload=436993]: Got next segment. Inline: false Dec 26 09:22:39.145927702 (497h53m39.017010737s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:22:39.145931580 (497h53m39.017014625s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:22:39.388524600 (497h53m39.259607645s) [seg_pos={0 0}, upload=436993]: Begin upload segment... Dec 26 09:22:39.388594132 (497h53m39.259677167s) [seg_pos={0 0}, upload=436993]: Done begin upload segment. Dec 26 09:22:39.388632365 (497h53m39.25971541s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.388653324 (497h53m39.259736359s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389447877 (497h53m39.260530922s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389850545 (497h53m39.26093358s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390368431 (497h53m39.261451476s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390681477 (497h53m39.261764512s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390976849 (497h53m39.262059884s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.392704890 (497h53m39.263787935s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.394668378 (497h53m39.265751413s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.396705798 (497h53m39.267788833s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.399751377 (497h53m39.270834412s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.400608700 (497h53m39.271691745s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.403933580 (497h53m39.275016626s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406414265 (497h53m39.2774973s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406579328 (497h53m39.277662374s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.408705978 (497h53m39.279789023s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:41.096910661 (497h53m40.967993696s) [upload=436993]: Next returned nil segment Dec 26 09:22:41.096923887 (497h53m40.968006922s) [upload=436993]: Waiting for error group managing segments... Dec 26 09:22:41.381072225 (497h53m41.25215526s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.383949273 (497h53m41.255032308s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.396690755 (497h53m41.26777379s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.411478754 (497h53m41.282561799s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421077892 (497h53m41.292160938s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421372643 (497h53m41.292455688s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.434903529 (497h53m41.305986574s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.435526675 (497h53m41.30660972s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.438982786 (497h53m41.310065821s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.625051454 (497h53m41.496134489s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.646488653 (497h53m41.517571688s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.687221335 (497h53m41.55830438s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690272836 (497h53m41.561355871s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690292143 (497h53m41.561375188s) [seg_pos={0 0}, upload=436993]: Segment reached optimal threshold of 13 pieces. Dec 26 09:22:41.690533532 (497h53m41.561616577s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection Dec 26 09:22:41.690545716 (497h53m41.561628751s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690586363 (497h53m41.561669398s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection Dec 26 09:22:41.690602573 (497h53m41.561685618s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690748803 (497h53m41.561831848s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 09:22:41.690758992 (497h53m41.561842037s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690901523 (497h53m41.561984568s) [seg_pos={0 0}, upload=436993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:22:41.691053442 (497h53m41.562136487s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:22:41.691060506 (497h53m41.562143551s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:22:41.976817686 (497h53m41.847900731s) [upload=436993]: Done uploading segments... Dec 26 09:22:41.976839939 (497h53m41.847922984s) [upload=436993]: Upload finished. err: <nil> Duration: 5.959276532s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXX....................X....XXX..X.......X.....XX.XX.XXX....] duration: 6.590385262s trace: https://jaeger.dev.storj.io/trace/389ac4bc3ff91e5c FAIL (2025-12-26 09:24:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding segments... Dec 26 09:22:39.145892195 (497h53m39.01697523s) [upload=436993]: Got next segment. Inline: false Dec 26 09:22:39.145927702 (497h53m39.017010737s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:22:39.145931580 (497h53m39.017014625s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:22:39.388524600 (497h53m39.259607645s) [seg_pos={0 0}, upload=436993]: Begin upload segment... Dec 26 09:22:39.388594132 (497h53m39.259677167s) [seg_pos={0 0}, upload=436993]: Done begin upload segment. Dec 26 09:22:39.388632365 (497h53m39.25971541s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.388653324 (497h53m39.259736359s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389447877 (497h53m39.260530922s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389850545 (497h53m39.26093358s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390368431 (497h53m39.261451476s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390681477 (497h53m39.261764512s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390976849 (497h53m39.262059884s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.392704890 (497h53m39.263787935s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.394668378 (497h53m39.265751413s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.396705798 (497h53m39.267788833s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.399751377 (497h53m39.270834412s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.400608700 (497h53m39.271691745s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.403933580 (497h53m39.275016626s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406414265 (497h53m39.2774973s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406579328 (497h53m39.277662374s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.408705978 (497h53m39.279789023s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:41.096910661 (497h53m40.967993696s) [upload=436993]: Next returned nil segment Dec 26 09:22:41.096923887 (497h53m40.968006922s) [upload=436993]: Waiting for error group managing segments... Dec 26 09:22:41.381072225 (497h53m41.25215526s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.383949273 (497h53m41.255032308s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.396690755 (497h53m41.26777379s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.411478754 (497h53m41.282561799s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421077892 (497h53m41.292160938s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421372643 (497h53m41.292455688s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.434903529 (497h53m41.305986574s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.435526675 (497h53m41.30660972s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.438982786 (497h53m41.310065821s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.625051454 (497h53m41.496134489s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.646488653 (497h53m41.517571688s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.687221335 (497h53m41.55830438s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690272836 (497h53m41.561355871s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690292143 (497h53m41.561375188s) [seg_pos={0 0}, upload=436993]: Segment reached optimal threshold of 13 pieces. Dec 26 09:22:41.690533532 (497h53m41.561616577s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection Dec 26 09:22:41.690545716 (497h53m41.561628751s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690586363 (497h53m41.561669398s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection Dec 26 09:22:41.690602573 (497h53m41.561685618s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690748803 (497h53m41.561831848s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 09:22:41.690758992 (497h53m41.561842037s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690901523 (497h53m41.561984568s) [seg_pos={0 0}, upload=436993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:22:41.691053442 (497h53m41.562136487s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:22:41.691060506 (497h53m41.562143551s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:22:41.976817686 (497h53m41.847900731s) [upload=436993]: Done uploading segments... Dec 26 09:22:41.976839939 (497h53m41.847922984s) [upload=436993]: Upload finished. err: <nil> Duration: 6.590414629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XXX....................X....XXX..X.......X.....XX.XX.XXX...] duration: 6.136146383s trace: https://jaeger.dev.storj.io/trace/389ac4bc3ff91e5c FAIL (2025-12-26 09:24:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding segments... Dec 26 09:22:39.145892195 (497h53m39.01697523s) [upload=436993]: Got next segment. Inline: false Dec 26 09:22:39.145927702 (497h53m39.017010737s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:22:39.145931580 (497h53m39.017014625s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:22:39.388524600 (497h53m39.259607645s) [seg_pos={0 0}, upload=436993]: Begin upload segment... Dec 26 09:22:39.388594132 (497h53m39.259677167s) [seg_pos={0 0}, upload=436993]: Done begin upload segment. Dec 26 09:22:39.388632365 (497h53m39.25971541s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.388653324 (497h53m39.259736359s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389447877 (497h53m39.260530922s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389850545 (497h53m39.26093358s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390368431 (497h53m39.261451476s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390681477 (497h53m39.261764512s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390976849 (497h53m39.262059884s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.392704890 (497h53m39.263787935s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.394668378 (497h53m39.265751413s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.396705798 (497h53m39.267788833s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.399751377 (497h53m39.270834412s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.400608700 (497h53m39.271691745s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.403933580 (497h53m39.275016626s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406414265 (497h53m39.2774973s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406579328 (497h53m39.277662374s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.408705978 (497h53m39.279789023s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:41.096910661 (497h53m40.967993696s) [upload=436993]: Next returned nil segment Dec 26 09:22:41.096923887 (497h53m40.968006922s) [upload=436993]: Waiting for error group managing segments... Dec 26 09:22:41.381072225 (497h53m41.25215526s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.383949273 (497h53m41.255032308s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.396690755 (497h53m41.26777379s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.411478754 (497h53m41.282561799s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421077892 (497h53m41.292160938s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421372643 (497h53m41.292455688s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.434903529 (497h53m41.305986574s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.435526675 (497h53m41.30660972s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.438982786 (497h53m41.310065821s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.625051454 (497h53m41.496134489s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.646488653 (497h53m41.517571688s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.687221335 (497h53m41.55830438s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690272836 (497h53m41.561355871s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690292143 (497h53m41.561375188s) [seg_pos={0 0}, upload=436993]: Segment reached optimal threshold of 13 pieces. Dec 26 09:22:41.690533532 (497h53m41.561616577s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection Dec 26 09:22:41.690545716 (497h53m41.561628751s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690586363 (497h53m41.561669398s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection Dec 26 09:22:41.690602573 (497h53m41.561685618s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690748803 (497h53m41.561831848s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 09:22:41.690758992 (497h53m41.561842037s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690901523 (497h53m41.561984568s) [seg_pos={0 0}, upload=436993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:22:41.691053442 (497h53m41.562136487s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:22:41.691060506 (497h53m41.562143551s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:22:41.976817686 (497h53m41.847900731s) [upload=436993]: Done uploading segments... Dec 26 09:22:41.976839939 (497h53m41.847922984s) [upload=436993]: Upload finished. err: <nil> Duration: 6.136177272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.XXX....................X....XXX..X.......X.....XX.XX.XXX..] duration: 6.062871796s trace: https://jaeger.dev.storj.io/trace/389ac4bc3ff91e5c FAIL (2025-12-26 09:23:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding segments... Dec 26 09:22:39.145892195 (497h53m39.01697523s) [upload=436993]: Got next segment. Inline: false Dec 26 09:22:39.145927702 (497h53m39.017010737s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:22:39.145931580 (497h53m39.017014625s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:22:39.388524600 (497h53m39.259607645s) [seg_pos={0 0}, upload=436993]: Begin upload segment... Dec 26 09:22:39.388594132 (497h53m39.259677167s) [seg_pos={0 0}, upload=436993]: Done begin upload segment. Dec 26 09:22:39.388632365 (497h53m39.25971541s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.388653324 (497h53m39.259736359s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389447877 (497h53m39.260530922s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389850545 (497h53m39.26093358s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390368431 (497h53m39.261451476s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390681477 (497h53m39.261764512s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390976849 (497h53m39.262059884s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.392704890 (497h53m39.263787935s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.394668378 (497h53m39.265751413s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.396705798 (497h53m39.267788833s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.399751377 (497h53m39.270834412s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.400608700 (497h53m39.271691745s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.403933580 (497h53m39.275016626s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406414265 (497h53m39.2774973s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406579328 (497h53m39.277662374s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.408705978 (497h53m39.279789023s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:41.096910661 (497h53m40.967993696s) [upload=436993]: Next returned nil segment Dec 26 09:22:41.096923887 (497h53m40.968006922s) [upload=436993]: Waiting for error group managing segments... Dec 26 09:22:41.381072225 (497h53m41.25215526s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.383949273 (497h53m41.255032308s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.396690755 (497h53m41.26777379s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.411478754 (497h53m41.282561799s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421077892 (497h53m41.292160938s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421372643 (497h53m41.292455688s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.434903529 (497h53m41.305986574s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.435526675 (497h53m41.30660972s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.438982786 (497h53m41.310065821s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.625051454 (497h53m41.496134489s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.646488653 (497h53m41.517571688s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.687221335 (497h53m41.55830438s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690272836 (497h53m41.561355871s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690292143 (497h53m41.561375188s) [seg_pos={0 0}, upload=436993]: Segment reached optimal threshold of 13 pieces. Dec 26 09:22:41.690533532 (497h53m41.561616577s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection Dec 26 09:22:41.690545716 (497h53m41.561628751s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690586363 (497h53m41.561669398s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection Dec 26 09:22:41.690602573 (497h53m41.561685618s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690748803 (497h53m41.561831848s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 09:22:41.690758992 (497h53m41.561842037s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690901523 (497h53m41.561984568s) [seg_pos={0 0}, upload=436993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:22:41.691053442 (497h53m41.562136487s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:22:41.691060506 (497h53m41.562143551s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:22:41.976817686 (497h53m41.847900731s) [upload=436993]: Done uploading segments... Dec 26 09:22:41.976839939 (497h53m41.847922984s) [upload=436993]: Upload finished. err: <nil> Duration: 6.062902484s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.XXX....................X....XXX..X.......X.....XX.XX.XXX.] duration: 6.052775902s trace: https://jaeger.dev.storj.io/trace/389ac4bc3ff91e5c FAIL (2025-12-26 09:23:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding segments... Dec 26 09:22:39.145892195 (497h53m39.01697523s) [upload=436993]: Got next segment. Inline: false Dec 26 09:22:39.145927702 (497h53m39.017010737s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:22:39.145931580 (497h53m39.017014625s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:22:39.388524600 (497h53m39.259607645s) [seg_pos={0 0}, upload=436993]: Begin upload segment... Dec 26 09:22:39.388594132 (497h53m39.259677167s) [seg_pos={0 0}, upload=436993]: Done begin upload segment. Dec 26 09:22:39.388632365 (497h53m39.25971541s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.388653324 (497h53m39.259736359s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389447877 (497h53m39.260530922s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389850545 (497h53m39.26093358s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390368431 (497h53m39.261451476s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390681477 (497h53m39.261764512s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390976849 (497h53m39.262059884s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.392704890 (497h53m39.263787935s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.394668378 (497h53m39.265751413s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.396705798 (497h53m39.267788833s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.399751377 (497h53m39.270834412s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.400608700 (497h53m39.271691745s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.403933580 (497h53m39.275016626s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406414265 (497h53m39.2774973s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406579328 (497h53m39.277662374s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.408705978 (497h53m39.279789023s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:41.096910661 (497h53m40.967993696s) [upload=436993]: Next returned nil segment Dec 26 09:22:41.096923887 (497h53m40.968006922s) [upload=436993]: Waiting for error group managing segments... Dec 26 09:22:41.381072225 (497h53m41.25215526s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.383949273 (497h53m41.255032308s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.396690755 (497h53m41.26777379s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.411478754 (497h53m41.282561799s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421077892 (497h53m41.292160938s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421372643 (497h53m41.292455688s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.434903529 (497h53m41.305986574s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.435526675 (497h53m41.30660972s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.438982786 (497h53m41.310065821s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.625051454 (497h53m41.496134489s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.646488653 (497h53m41.517571688s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.687221335 (497h53m41.55830438s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690272836 (497h53m41.561355871s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690292143 (497h53m41.561375188s) [seg_pos={0 0}, upload=436993]: Segment reached optimal threshold of 13 pieces. Dec 26 09:22:41.690533532 (497h53m41.561616577s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection Dec 26 09:22:41.690545716 (497h53m41.561628751s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690586363 (497h53m41.561669398s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection Dec 26 09:22:41.690602573 (497h53m41.561685618s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690748803 (497h53m41.561831848s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 09:22:41.690758992 (497h53m41.561842037s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690901523 (497h53m41.561984568s) [seg_pos={0 0}, upload=436993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:22:41.691053442 (497h53m41.562136487s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:22:41.691060506 (497h53m41.562143551s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:22:41.976817686 (497h53m41.847900731s) [upload=436993]: Done uploading segments... Dec 26 09:22:41.976839939 (497h53m41.847922984s) [upload=436993]: Upload finished. err: <nil> Duration: 6.052808184s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.XXX....................X....XXX..X.......X.....XX.XX.XXX] duration: 30.003004703s trace: https://jaeger.dev.storj.io/trace/389ac4bc3ff91e5c FAIL (2025-12-26 09:23:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding segments... Dec 26 09:22:39.145892195 (497h53m39.01697523s) [upload=436993]: Got next segment. Inline: false Dec 26 09:22:39.145927702 (497h53m39.017010737s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:22:39.145931580 (497h53m39.017014625s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:22:39.388524600 (497h53m39.259607645s) [seg_pos={0 0}, upload=436993]: Begin upload segment... Dec 26 09:22:39.388594132 (497h53m39.259677167s) [seg_pos={0 0}, upload=436993]: Done begin upload segment. Dec 26 09:22:39.388632365 (497h53m39.25971541s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.388653324 (497h53m39.259736359s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389447877 (497h53m39.260530922s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389850545 (497h53m39.26093358s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390368431 (497h53m39.261451476s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390681477 (497h53m39.261764512s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390976849 (497h53m39.262059884s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.392704890 (497h53m39.263787935s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.394668378 (497h53m39.265751413s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.396705798 (497h53m39.267788833s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.399751377 (497h53m39.270834412s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.400608700 (497h53m39.271691745s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.403933580 (497h53m39.275016626s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406414265 (497h53m39.2774973s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406579328 (497h53m39.277662374s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.408705978 (497h53m39.279789023s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:41.096910661 (497h53m40.967993696s) [upload=436993]: Next returned nil segment Dec 26 09:22:41.096923887 (497h53m40.968006922s) [upload=436993]: Waiting for error group managing segments... Dec 26 09:22:41.381072225 (497h53m41.25215526s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.383949273 (497h53m41.255032308s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.396690755 (497h53m41.26777379s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.411478754 (497h53m41.282561799s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421077892 (497h53m41.292160938s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421372643 (497h53m41.292455688s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.434903529 (497h53m41.305986574s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.435526675 (497h53m41.30660972s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.438982786 (497h53m41.310065821s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.625051454 (497h53m41.496134489s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.646488653 (497h53m41.517571688s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.687221335 (497h53m41.55830438s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690272836 (497h53m41.561355871s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690292143 (497h53m41.561375188s) [seg_pos={0 0}, upload=436993]: Segment reached optimal threshold of 13 pieces. Dec 26 09:22:41.690533532 (497h53m41.561616577s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection Dec 26 09:22:41.690545716 (497h53m41.561628751s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690586363 (497h53m41.561669398s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection Dec 26 09:22:41.690602573 (497h53m41.561685618s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690748803 (497h53m41.561831848s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 09:22:41.690758992 (497h53m41.561842037s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690901523 (497h53m41.561984568s) [seg_pos={0 0}, upload=436993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:22:41.691053442 (497h53m41.562136487s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:22:41.691060506 (497h53m41.562143551s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:22:41.976817686 (497h53m41.847900731s) [upload=436993]: Done uploading segments... Dec 26 09:22:41.976839939 (497h53m41.847922984s) [upload=436993]: Upload finished. err: <nil> Duration: 30.00302881s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X..X.XXX....................X....XXX..X.......X.....XX.XX.XX] duration: 30.002325463s trace: https://jaeger.dev.storj.io/trace/635edce129cc0db7 FAIL (2025-12-26 09:22:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: VFRPHFVFFUSGYPRN7VI3S2BLHVVIIABYRLDXRNUWV4JAW6WXOB3Q) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: AW3EWNN4BZ4NC5WSVXQ2O6NL5O6GIITPO2JZQTN2JCC4EISJV3LA) context deadline exceeded ======LOGS====== 26 09:21:46.403461474 (497h52m46.274544519s) [upload=436981]: Starting upload Dec 26 09:21:46.403511469 (497h52m46.274594514s) [upload=436981]: Done starting upload Dec 26 09:21:46.403858680 (497h52m46.274941715s) [upload=436981]: Uploading segments... Dec 26 09:21:46.403919727 (497h52m46.275002762s) [upload=436981]: Got next segment. Inline: false Dec 26 09:21:46.403948922 (497h52m46.275031957s) [upload=436981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:21:46.403952960 (497h52m46.275035995s) [upload=436981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:21:46.647306216 (497h52m46.518389251s) [seg_pos={0 0}, upload=436981]: Begin upload segment... Dec 26 09:21:46.647349459 (497h52m46.518432494s) [seg_pos={0 0}, upload=436981]: Done begin upload segment. Dec 26 09:21:46.647402579 (497h52m46.518485625s) [noise=true , address=145.79.227.43:4159 , piece_id=5FID3F3OPXSKHYLCO3TZ7VQISKSDAXA7ACIECDQ627NB4AARCN3Q, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.647412368 (497h52m46.518495403s) [noise=true , address=77.90.31.11:4556 , piece_id=KR2ZCHXOSE5R6H2CM5WBN24MROZGJRYRXLYYD7H4BWW66AF6FMJA, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.647572744 (497h52m46.518655779s) [noise=true , address=45.91.116.5:4300 , piece_id=AW3EWNN4BZ4NC5WSVXQ2O6NL5O6GIITPO2JZQTN2JCC4EISJV3LA, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.648030816 (497h52m46.519113851s) [noise=true , address=145.79.227.24:4583 , piece_id=2AZ2HYGMB25UKP2JPEO6G3CC7MMPUVBWUAYXJDU3XDU5GER4EITQ, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.648362508 (497h52m46.519445543s) [noise=true , address=45.91.116.8:4047 , piece_id=5XTTMLK4OXO65TPGLQPJB4KUC6FPFUSJQYTOCOIGM3DABKIBMBLA, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.648696142 (497h52m46.519779177s) [noise=true , address=145.79.227.16:4910 , piece_id=YM5S6PC5XJZUL2AJADKIQ7RFWQTDSPHKAYU5WUWWW35CIDW6CN5A, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.648950698 (497h52m46.520033733s) [noise=true , address=45.91.116.9:4261 , piece_id=ZHHYGQ66RFKTSY7N2JOMNRRJDNO4ZYGARDD423VMQZIHQCFC2GVQ, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.649234328 (497h52m46.520317373s) [noise=true , address=145.79.227.27:4690 , piece_id=D72AGER7VQXQEGLBMXOOIJLLV5A5P5VZT5V6RO6XGSKJ342E5WQA, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.649552293 (497h52m46.520635328s) [noise=true , address=145.79.227.36:4517 , piece_id=DDT2B6DTVIHDBN6ZLZI7LMSHHNLS2P5XXKEYDDMIMWY7HUD7WPEQ, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.649898542 (497h52m46.520981578s) [noise=true , address=145.79.227.11:4732 , piece_id=KQUVJXZWSA2QOX56ACRPJ73O2WKHB4SVQWYVDWTZ5SKRBYEDGNBQ, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.650004073 (497h52m46.521087108s) [noise=true , address=145.79.227.33:4870 , piece_id=QWGK2HCF2O4H6URGUENDGJ4X46XDJRZ5QHGDUPQUGGCO3BUPD54Q, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.650441406 (497h52m46.521524451s) [noise=true , address=45.91.116.13:4154 , piece_id=VFRPHFVFFUSGYPRN7VI3S2BLHVVIIABYRLDXRNUWV4JAW6WXOB3Q, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.650701972 (497h52m46.521785007s) [noise=true , address=45.91.116.4:4940 , piece_id=OJWDYUEOWDKNW5ZFEMPVAX2VTHUOVWIBOUY6MVVHYGUIHTVRKAAA, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.650756926 (497h52m46.521839961s) [noise=true , address=145.79.227.4:4186 , piece_id=HKFOMWU56YFWNXUDIFEVLCZO2I6TP76E667KSV5CS5V57XHROR5A, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.651049103 (497h52m46.522132138s) [noise=true , address=77.90.31.21:4880 , piece_id=WKAMWYGSYFX2OSONUTTQNXYGRRBWLI2TFVFB6OCG4JGLFFLXKXTQ, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.651396695 (497h52m46.52247973s) [noise=true , address=145.79.227.50:4373 , piece_id=P56Y6NXR56TU2S6XN5J37VIKZ4XPCM6DXQO7HDQKLGS4UFDZHFXQ, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:48.264473462 (497h52m48.135556497s) [upload=436981]: Next returned nil segment Dec 26 09:21:48.264487469 (497h52m48.135570514s) [upload=436981]: Waiting for error group managing segments... Dec 26 09:21:48.653456122 (497h52m48.524539157s) [noise=true , address=145.79.227.16:4910 , piece_id=YM5S6PC5XJZUL2AJADKIQ7RFWQTDSPHKAYU5WUWWW35CIDW6CN5A, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.653597572 (497h52m48.524680617s) [noise=true , address=145.79.227.24:4583 , piece_id=2AZ2HYGMB25UKP2JPEO6G3CC7MMPUVBWUAYXJDU3XDU5GER4EITQ, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.660055890 (497h52m48.531138925s) [noise=true , address=145.79.227.36:4517 , piece_id=DDT2B6DTVIHDBN6ZLZI7LMSHHNLS2P5XXKEYDDMIMWY7HUD7WPEQ, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.667083301 (497h52m48.538166336s) [noise=true , address=145.79.227.4:4186 , piece_id=HKFOMWU56YFWNXUDIFEVLCZO2I6TP76E667KSV5CS5V57XHROR5A, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.672307369 (497h52m48.543390414s) [noise=true , address=145.79.227.50:4373 , piece_id=P56Y6NXR56TU2S6XN5J37VIKZ4XPCM6DXQO7HDQKLGS4UFDZHFXQ, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.675114244 (497h52m48.546197279s) [noise=true , address=145.79.227.27:4690 , piece_id=D72AGER7VQXQEGLBMXOOIJLLV5A5P5VZT5V6RO6XGSKJ342E5WQA, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.678730930 (497h52m48.549813965s) [noise=true , address=145.79.227.11:4732 , piece_id=KQUVJXZWSA2QOX56ACRPJ73O2WKHB4SVQWYVDWTZ5SKRBYEDGNBQ, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.784465517 (497h52m48.655548552s) [noise=true , address=145.79.227.43:4159 , piece_id=5FID3F3OPXSKHYLCO3TZ7VQISKSDAXA7ACIECDQ627NB4AARCN3Q, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.905033629 (497h52m48.776116664s) [noise=true , address=145.79.227.33:4870 , piece_id=QWGK2HCF2O4H6URGUENDGJ4X46XDJRZ5QHGDUPQUGGCO3BUPD54Q, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.919792152 (497h52m48.790875197s) [noise=true , address=45.91.116.5:4300 , piece_id=AW3EWNN4BZ4NC5WSVXQ2O6NL5O6GIITPO2JZQTN2JCC4EISJV3LA, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:49.024066258 (497h52m48.895149304s) [noise=true , address=77.90.31.11:4556 , piece_id=KR2ZCHXOSE5R6H2CM5WBN24MROZGJRYRXLYYD7H4BWW66AF6FMJA, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:49.048193240 (497h52m48.919276285s) [noise=true , address=45.91.116.4:4940 , piece_id=OJWDYUEOWDKNW5ZFEMPVAX2VTHUOVWIBOUY6MVVHYGUIHTVRKAAA, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:49.763601441 (497h52m49.634684476s) [noise=true , address=45.91.116.13:4154 , piece_id=VFRPHFVFFUSGYPRN7VI3S2BLHVVIIABYRLDXRNUWV4JAW6WXOB3Q, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:49.763623944 (497h52m49.634706979s) [seg_pos={0 0}, upload=436981]: Segment reached optimal threshold of 13 pieces. Dec 26 09:21:49.763887766 (497h52m49.634970801s) [noise=true , address=77.90.31.21:4880 , piece_id=WKAMWYGSYFX2OSONUTTQNXYGRRBWLI2TFVFB6OCG4JGLFFLXKXTQ, seg_pos={0 0}, upload=436981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:46612->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:46612->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46612->77.90.31.21:4880: use of closed network connection Dec 26 09:21:49.763896643 (497h52m49.634979688s) [noise=true , address=77.90.31.21:4880 , piece_id=WKAMWYGSYFX2OSONUTTQNXYGRRBWLI2TFVFB6OCG4JGLFFLXKXTQ, seg_pos={0 0}, upload=436981]: Long tail cancellation due to optimal threshold Dec 26 09:21:49.763958261 (497h52m49.635041306s) [noise=true , address=45.91.116.9:4261 , piece_id=ZHHYGQ66RFKTSY7N2JOMNRRJDNO4ZYGARDD423VMQZIHQCFC2GVQ, seg_pos={0 0}, upload=436981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 09:21:49.763968891 (497h52m49.635051956s) [noise=true , address=45.91.116.9:4261 , piece_id=ZHHYGQ66RFKTSY7N2JOMNRRJDNO4ZYGARDD423VMQZIHQCFC2GVQ, seg_pos={0 0}, upload=436981]: Long tail cancellation due to optimal threshold Dec 26 09:21:49.764188899 (497h52m49.635271935s) [noise=true , address=45.91.116.8:4047 , piece_id=5XTTMLK4OXO65TPGLQPJB4KUC6FPFUSJQYTOCOIGM3DABKIBMBLA, seg_pos={0 0}, upload=436981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 09:21:49.764198507 (497h52m49.635281553s) [noise=true , address=45.91.116.8:4047 , piece_id=5XTTMLK4OXO65TPGLQPJB4KUC6FPFUSJQYTOCOIGM3DABKIBMBLA, seg_pos={0 0}, upload=436981]: Long tail cancellation due to optimal threshold Dec 26 09:21:49.764222283 (497h52m49.635305328s) [seg_pos={0 0}, upload=436981]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:21:49.764327614 (497h52m49.635410649s) [upload=436981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:21:49.764334797 (497h52m49.635417832s) [upload=436981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:21:50.061515470 (497h52m49.932598505s) [upload=436981]: Done uploading segments... Dec 26 09:21:50.061533525 (497h52m49.93261656s) [upload=436981]: Upload finished. err: <nil> Duration: 30.002349898s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX..X.XXX....................X....XXX..X.......X.....XX.XX.X] duration: 30.003183353s trace: https://jaeger.dev.storj.io/trace/4f8203c617f957b1 FAIL (2025-12-26 09:21:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 25RBE2EHWTYANKZ43DNGOA5DY557NEQYAWLZLFHWQJ7DFE4H43MQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 2ND2LKYAJ524K6BEO5TU3KGQHJ6ZF7MZT2JJ2ES3VWSUTCGBHX4Q) context deadline exceeded ======LOGS====== 97h51m46.506041203s) [upload=436968]: Done starting upload Dec 26 09:20:46.635092524 (497h51m46.506175559s) [upload=436968]: Uploading segments... Dec 26 09:20:46.635257146 (497h51m46.506340181s) [upload=436968]: Got next segment. Inline: false Dec 26 09:20:46.635303966 (497h51m46.506387001s) [upload=436968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:20:46.635310869 (497h51m46.506393904s) [upload=436968]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:20:46.880233915 (497h51m46.75131696s) [seg_pos={0 0}, upload=436968]: Begin upload segment... Dec 26 09:20:46.880298448 (497h51m46.751381493s) [seg_pos={0 0}, upload=436968]: Done begin upload segment. Dec 26 09:20:46.880376086 (497h51m46.751459121s) [noise=true , address=45.91.116.5:4300 , piece_id=RHDK5XINIGJSU33ZL7U45RP537ECE5QFC6TUCSI7FHP4XPYP4OPQ, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.880412675 (497h51m46.75149572s) [noise=true , address=145.79.227.45:4230 , piece_id=NVPE4OQRKKHBN2TOX475UJSDWUHTTFTBFISKKIOQJP6BNYHI4G2Q, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.880923558 (497h51m46.752006593s) [noise=true , address=145.79.227.46:4601 , piece_id=4Y4E3NFZT5CL3A4X23O4OJLH2QZD7HHC74KN3ON6LV37KHT2MXPA, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.881327978 (497h51m46.752411013s) [noise=true , address=145.79.227.15:4875 , piece_id=DTEYPG3RIGL3ADEINNI7AQIFD2YZ52KPXYRFZ4PV4FXTWCNXXTFA, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.881738971 (497h51m46.752822016s) [noise=true , address=45.91.116.7:4012 , piece_id=BV7LPZTAJYJ3F2PDHYOYSR7TYPPZGAUT5VTT6BCQYGHSD64FUAZA, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.882266004 (497h51m46.753349049s) [noise=true , address=145.79.227.41:4085 , piece_id=PK2GU7N3ANQ3SUTRWECXO3FEKLQE53KCUUDYRVMTL5H4LQ7RYNXQ, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.882699660 (497h51m46.753782695s) [noise=true , address=145.79.227.19:4079 , piece_id=7G6HP436N5QNWRT56S4HVJ4UFFGKEM4WY3L42TJNY57CHQ3GHRFA, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.883123627 (497h51m46.754206673s) [noise=true , address=145.79.227.36:4519 , piece_id=OKPLE32ZHCGAXG6RXCKLS7EU6QDYRDOW24FLVNI5XSZ356GXOH4A, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.883646082 (497h51m46.754729127s) [noise=true , address=45.91.116.9:4264 , piece_id=2ND2LKYAJ524K6BEO5TU3KGQHJ6ZF7MZT2JJ2ES3VWSUTCGBHX4Q, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.884950786 (497h51m46.756033821s) [noise=true , address=145.79.227.2:4113 , piece_id=LCE2WXLLK47ZNIBRDV6PWRLSEVN3Y35HKROO3M73FH2QU3YDZH4Q, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.885235760 (497h51m46.756318805s) [noise=true , address=145.79.227.17:4971 , piece_id=O4RUOKWECAN6X46WFMLFMO47WKIUOP6C33I6XFDIAO3ZI2F7NU7A, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.886712912 (497h51m46.757795957s) [noise=true , address=45.91.116.13:4155 , piece_id=25RBE2EHWTYANKZ43DNGOA5DY557NEQYAWLZLFHWQJ7DFE4H43MQ, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.886935636 (497h51m46.758018681s) [noise=true , address=45.91.116.10:4083 , piece_id=WF73W5YJ2GGPUCSPOQJNSH4RRMLCMFM6DCEF46WQNKXEKGV5H43A, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.888667403 (497h51m46.759750439s) [noise=true , address=77.90.31.13:4629 , piece_id=JMUD2VDIJI42NZNLHY7ZCLLPYR6ZZRFTVOPCIPRG4WTPBFNFFXAQ, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.891870473 (497h51m46.762953518s) [noise=true , address=145.79.227.49:4339 , piece_id=AFZSOFD5ODAEYJXHEAXCY4YZ7UV63J7JDNJG5BZJGZ2JD3ZP5SOA, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.895627537 (497h51m46.766710582s) [noise=true , address=45.91.116.14:4192 , piece_id=3Y6WEL7OSABCDL6G5C3HA53NAA6JWTIZACXJNWF36F2NKHWZNE7Q, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:48.610310399 (497h51m48.481393434s) [upload=436968]: Next returned nil segment Dec 26 09:20:48.610332281 (497h51m48.481415316s) [upload=436968]: Waiting for error group managing segments... Dec 26 09:20:48.897403625 (497h51m48.76848667s) [noise=true , address=145.79.227.2:4113 , piece_id=LCE2WXLLK47ZNIBRDV6PWRLSEVN3Y35HKROO3M73FH2QU3YDZH4Q, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:48.898371066 (497h51m48.769454111s) [noise=true , address=145.79.227.46:4601 , piece_id=4Y4E3NFZT5CL3A4X23O4OJLH2QZD7HHC74KN3ON6LV37KHT2MXPA, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:48.898631402 (497h51m48.769714447s) [noise=true , address=145.79.227.19:4079 , piece_id=7G6HP436N5QNWRT56S4HVJ4UFFGKEM4WY3L42TJNY57CHQ3GHRFA, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:48.899402430 (497h51m48.770485465s) [noise=true , address=145.79.227.41:4085 , piece_id=PK2GU7N3ANQ3SUTRWECXO3FEKLQE53KCUUDYRVMTL5H4LQ7RYNXQ, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:48.901736314 (497h51m48.772819349s) [noise=true , address=145.79.227.15:4875 , piece_id=DTEYPG3RIGL3ADEINNI7AQIFD2YZ52KPXYRFZ4PV4FXTWCNXXTFA, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:48.905576196 (497h51m48.776659231s) [noise=true , address=145.79.227.45:4230 , piece_id=NVPE4OQRKKHBN2TOX475UJSDWUHTTFTBFISKKIOQJP6BNYHI4G2Q, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:48.911288242 (497h51m48.782371287s) [noise=true , address=145.79.227.36:4519 , piece_id=OKPLE32ZHCGAXG6RXCKLS7EU6QDYRDOW24FLVNI5XSZ356GXOH4A, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:48.912763823 (497h51m48.783846858s) [noise=true , address=145.79.227.49:4339 , piece_id=AFZSOFD5ODAEYJXHEAXCY4YZ7UV63J7JDNJG5BZJGZ2JD3ZP5SOA, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:48.913918571 (497h51m48.785001606s) [noise=true , address=145.79.227.17:4971 , piece_id=O4RUOKWECAN6X46WFMLFMO47WKIUOP6C33I6XFDIAO3ZI2F7NU7A, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:49.151944564 (497h51m49.023027609s) [noise=true , address=45.91.116.13:4155 , piece_id=25RBE2EHWTYANKZ43DNGOA5DY557NEQYAWLZLFHWQJ7DFE4H43MQ, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:49.182025213 (497h51m49.053108248s) [noise=true , address=45.91.116.5:4300 , piece_id=RHDK5XINIGJSU33ZL7U45RP537ECE5QFC6TUCSI7FHP4XPYP4OPQ, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:49.268589917 (497h51m49.139672962s) [noise=true , address=45.91.116.9:4264 , piece_id=2ND2LKYAJ524K6BEO5TU3KGQHJ6ZF7MZT2JJ2ES3VWSUTCGBHX4Q, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:49.372518365 (497h51m49.2436014s) [noise=true , address=45.91.116.14:4192 , piece_id=3Y6WEL7OSABCDL6G5C3HA53NAA6JWTIZACXJNWF36F2NKHWZNE7Q, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:49.372542651 (497h51m49.243625696s) [seg_pos={0 0}, upload=436968]: Segment reached optimal threshold of 13 pieces. Dec 26 09:20:49.372783179 (497h51m49.243866214s) [noise=true , address=77.90.31.13:4629 , piece_id=JMUD2VDIJI42NZNLHY7ZCLLPYR6ZZRFTVOPCIPRG4WTPBFNFFXAQ, seg_pos={0 0}, upload=436968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 26 09:20:49.372793038 (497h51m49.243876093s) [noise=true , address=77.90.31.13:4629 , piece_id=JMUD2VDIJI42NZNLHY7ZCLLPYR6ZZRFTVOPCIPRG4WTPBFNFFXAQ, seg_pos={0 0}, upload=436968]: Long tail cancellation due to optimal threshold Dec 26 09:20:49.373018356 (497h51m49.244101391s) [noise=true , address=45.91.116.10:4083 , piece_id=WF73W5YJ2GGPUCSPOQJNSH4RRMLCMFM6DCEF46WQNKXEKGV5H43A, seg_pos={0 0}, upload=436968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35020->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35020->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35020->45.91.116.10:4083: use of closed network connection Dec 26 09:20:49.373039357 (497h51m49.244122402s) [noise=true , address=45.91.116.10:4083 , piece_id=WF73W5YJ2GGPUCSPOQJNSH4RRMLCMFM6DCEF46WQNKXEKGV5H43A, seg_pos={0 0}, upload=436968]: Long tail cancellation due to optimal threshold Dec 26 09:20:49.373256840 (497h51m49.244339906s) [noise=true , address=45.91.116.7:4012 , piece_id=BV7LPZTAJYJ3F2PDHYOYSR7TYPPZGAUT5VTT6BCQYGHSD64FUAZA, seg_pos={0 0}, upload=436968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: read tcp 188.166.225.170:51752->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 26 09:20:49.373267041 (497h51m49.244350086s) [noise=true , address=45.91.116.7:4012 , piece_id=BV7LPZTAJYJ3F2PDHYOYSR7TYPPZGAUT5VTT6BCQYGHSD64FUAZA, seg_pos={0 0}, upload=436968]: Long tail cancellation due to optimal threshold Dec 26 09:20:49.373293250 (497h51m49.244376285s) [seg_pos={0 0}, upload=436968]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:20:49.373374585 (497h51m49.24445763s) [upload=436968]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:20:49.373380556 (497h51m49.244463602s) [upload=436968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:20:49.659951867 (497h51m49.531034903s) [upload=436968]: Done uploading segments... Dec 26 09:20:49.659970984 (497h51m49.531054019s) [upload=436968]: Upload finished. err: <nil> Duration: 30.003205645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 42.668587794s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:20:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 42.668613933s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)