Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.X...................X..................] duration: 25.199740273s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:51:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 25.199765491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X....................X.....X....X.X...X....X......] duration: 5.924872415s trace: https://jaeger.dev.storj.io/trace/10d2927d069b4ed1 FAIL (2025-12-28 22:51:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 22:46:16.281303410 (559h17m16.152386445s) [upload=490322]: Starting upload Dec 28 22:46:16.281379415 (559h17m16.15246246s) [upload=490322]: Done starting upload Dec 28 22:46:16.281492900 (559h17m16.152575945s) [upload=490322]: Uploading segments... Dec 28 22:46:16.281736144 (559h17m16.152819189s) [upload=490322]: Got next segment. Inline: false Dec 28 22:46:16.281827829 (559h17m16.152910874s) [upload=490322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:46:16.281833499 (559h17m16.152916534s) [upload=490322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:46:16.532054602 (559h17m16.403137637s) [seg_pos={0 0}, upload=490322]: Begin upload segment... Dec 28 22:46:16.532131298 (559h17m16.403214343s) [seg_pos={0 0}, upload=490322]: Done begin upload segment. Dec 28 22:46:16.532199558 (559h17m16.403282593s) [noise=true , address=45.91.116.8:4047 , piece_id=OE32NXK6YJHX7TG36HPAHM7CQ7Z43YQMGVHADNZJCID7IA3PESHQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.532247149 (559h17m16.403330184s) [noise=true , address=145.79.227.16:4910 , piece_id=MDA4MGAKXBMGBRQSZ7WC4OS3MRQQ3VFJ3W3TYHXJB3SCB25XYNFA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.532401703 (559h17m16.403484738s) [noise=true , address=145.79.227.29:4941 , piece_id=ONW7MMHOMTQWMAAKKW43YXNB53KOZOQUA3TYDBLVT6HCVNKDWAOA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533103579 (559h17m16.404186614s) [noise=true , address=145.79.227.42:4121 , piece_id=CALDXEHAPLIINBUJVB25J2Z6J6QHX3LVGVGNQ4YTTDZ2UMU2QSDA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533363634 (559h17m16.404446669s) [noise=true , address=145.79.227.3:4150 , piece_id=KCXQM5B6E3QKFVFBIDIKLBNE4YMU4RZAUKTUWQJFHEBZNC26E2SA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533498531 (559h17m16.404581576s) [noise=true , address=145.79.227.27:4691 , piece_id=7F373BDGJHBKITEIYRQ7QJCJRIHPRTDFWPZKKQPFTSJPVFOSJTQQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533939650 (559h17m16.405022685s) [noise=true , address=145.79.227.51:4410 , piece_id=5FX23RQJ5O65SKFTEW7BO2MP36VNI42RNYWUHE7PP4TZ4ZYSBYWQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534185308 (559h17m16.405268343s) [noise=true , address=145.79.227.36:4518 , piece_id=7TUGDNTYFP6AWE7ESKPDDU33ZBTUFOPTVMH4C5GRORO66DVTR65A, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534401649 (559h17m16.405484684s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534675090 (559h17m16.405758125s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534732289 (559h17m16.405815335s) [noise=true , address=145.79.227.37:4566 , piece_id=SASNC4CAPW2F5OUSZHSKRPOYUMMMSQ5G6KHZIW4NVFZM2RAEEZGQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535051748 (559h17m16.406134783s) [noise=true , address=145.79.227.11:4729 , piece_id=R25WH6YYDQ36YUIINIC3VNDEWESXMCO6K43YWE24EAQYCTELD6XA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535147068 (559h17m16.406230113s) [noise=true , address=45.91.116.13:4155 , piece_id=EDJCUZF5LE62KS3SDNRWMIY7UKXYP3JND7EXF3T5SX3OLKIHCLPA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535610190 (559h17m16.406693225s) [noise=true , address=145.79.227.38:4979 , piece_id=QYCUHBBHCQA2M4IEDWEA6TTGLU43FUKLQHSR2SQCMC4SHZNIJE3A, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.536199472 (559h17m16.407282517s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.536784084 (559h17m16.40786713s) [noise=true , address=45.91.116.5:4297 , piece_id=2MCIT75SXRVL5VE4CHCKRIDYWEDK54TA4ZTYF2ZQ4Z3FSDWIZ7MA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:18.203909763 (559h17m18.074992808s) [upload=490322]: Next returned nil segment Dec 28 22:46:18.203934080 (559h17m18.075017125s) [upload=490322]: Waiting for error group managing segments... Dec 28 22:46:18.533777259 (559h17m18.404860294s) [noise=true , address=145.79.227.16:4910 , piece_id=MDA4MGAKXBMGBRQSZ7WC4OS3MRQQ3VFJ3W3TYHXJB3SCB25XYNFA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.535160322 (559h17m18.406243367s) [noise=true , address=145.79.227.3:4150 , piece_id=KCXQM5B6E3QKFVFBIDIKLBNE4YMU4RZAUKTUWQJFHEBZNC26E2SA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.537196999 (559h17m18.408280044s) [noise=true , address=145.79.227.11:4729 , piece_id=R25WH6YYDQ36YUIINIC3VNDEWESXMCO6K43YWE24EAQYCTELD6XA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.548289885 (559h17m18.41937292s) [noise=true , address=145.79.227.51:4410 , piece_id=5FX23RQJ5O65SKFTEW7BO2MP36VNI42RNYWUHE7PP4TZ4ZYSBYWQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.548489425 (559h17m18.41957246s) [noise=true , address=145.79.227.42:4121 , piece_id=CALDXEHAPLIINBUJVB25J2Z6J6QHX3LVGVGNQ4YTTDZ2UMU2QSDA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.553480856 (559h17m18.424563891s) [noise=true , address=145.79.227.36:4518 , piece_id=7TUGDNTYFP6AWE7ESKPDDU33ZBTUFOPTVMH4C5GRORO66DVTR65A, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.554020704 (559h17m18.425103749s) [noise=true , address=145.79.227.37:4566 , piece_id=SASNC4CAPW2F5OUSZHSKRPOYUMMMSQ5G6KHZIW4NVFZM2RAEEZGQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.557268637 (559h17m18.428351672s) [noise=true , address=145.79.227.38:4979 , piece_id=QYCUHBBHCQA2M4IEDWEA6TTGLU43FUKLQHSR2SQCMC4SHZNIJE3A, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.561679774 (559h17m18.432762809s) [noise=true , address=145.79.227.27:4691 , piece_id=7F373BDGJHBKITEIYRQ7QJCJRIHPRTDFWPZKKQPFTSJPVFOSJTQQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.620428987 (559h17m18.491512022s) [noise=true , address=145.79.227.29:4941 , piece_id=ONW7MMHOMTQWMAAKKW43YXNB53KOZOQUA3TYDBLVT6HCVNKDWAOA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.734831199 (559h17m18.605914234s) [noise=true , address=45.91.116.5:4297 , piece_id=2MCIT75SXRVL5VE4CHCKRIDYWEDK54TA4ZTYF2ZQ4Z3FSDWIZ7MA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.764169667 (559h17m18.635252712s) [noise=true , address=45.91.116.13:4155 , piece_id=EDJCUZF5LE62KS3SDNRWMIY7UKXYP3JND7EXF3T5SX3OLKIHCLPA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.899179717 (559h17m18.770262752s) [noise=true , address=45.91.116.8:4047 , piece_id=OE32NXK6YJHX7TG36HPAHM7CQ7Z43YQMGVHADNZJCID7IA3PESHQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.899204645 (559h17m18.77028769s) [seg_pos={0 0}, upload=490322]: Segment reached optimal threshold of 13 pieces. Dec 28 22:46:18.899419233 (559h17m18.770502269s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899428721 (559h17m18.770511756s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899444932 (559h17m18.770527977s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899449651 (559h17m18.770532686s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899590659 (559h17m18.770673694s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899601921 (559h17m18.770684966s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899621057 (559h17m18.770704092s) [seg_pos={0 0}, upload=490322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:46:18.899821439 (559h17m18.770904474s) [upload=490322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:46:18.899827490 (559h17m18.770910535s) [upload=490322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:46:19.190656313 (559h17m19.061739348s) [upload=490322]: Done uploading segments... Dec 28 22:46:19.190698935 (559h17m19.06178197s) [upload=490322]: Upload finished. err: <nil> Duration: 5.924899005s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X....................X.....X....X.X...X....X.....] duration: 6.04314976s trace: https://jaeger.dev.storj.io/trace/10d2927d069b4ed1 FAIL (2025-12-28 22:50:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 22:46:16.281303410 (559h17m16.152386445s) [upload=490322]: Starting upload Dec 28 22:46:16.281379415 (559h17m16.15246246s) [upload=490322]: Done starting upload Dec 28 22:46:16.281492900 (559h17m16.152575945s) [upload=490322]: Uploading segments... Dec 28 22:46:16.281736144 (559h17m16.152819189s) [upload=490322]: Got next segment. Inline: false Dec 28 22:46:16.281827829 (559h17m16.152910874s) [upload=490322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:46:16.281833499 (559h17m16.152916534s) [upload=490322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:46:16.532054602 (559h17m16.403137637s) [seg_pos={0 0}, upload=490322]: Begin upload segment... Dec 28 22:46:16.532131298 (559h17m16.403214343s) [seg_pos={0 0}, upload=490322]: Done begin upload segment. Dec 28 22:46:16.532199558 (559h17m16.403282593s) [noise=true , address=45.91.116.8:4047 , piece_id=OE32NXK6YJHX7TG36HPAHM7CQ7Z43YQMGVHADNZJCID7IA3PESHQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.532247149 (559h17m16.403330184s) [noise=true , address=145.79.227.16:4910 , piece_id=MDA4MGAKXBMGBRQSZ7WC4OS3MRQQ3VFJ3W3TYHXJB3SCB25XYNFA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.532401703 (559h17m16.403484738s) [noise=true , address=145.79.227.29:4941 , piece_id=ONW7MMHOMTQWMAAKKW43YXNB53KOZOQUA3TYDBLVT6HCVNKDWAOA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533103579 (559h17m16.404186614s) [noise=true , address=145.79.227.42:4121 , piece_id=CALDXEHAPLIINBUJVB25J2Z6J6QHX3LVGVGNQ4YTTDZ2UMU2QSDA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533363634 (559h17m16.404446669s) [noise=true , address=145.79.227.3:4150 , piece_id=KCXQM5B6E3QKFVFBIDIKLBNE4YMU4RZAUKTUWQJFHEBZNC26E2SA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533498531 (559h17m16.404581576s) [noise=true , address=145.79.227.27:4691 , piece_id=7F373BDGJHBKITEIYRQ7QJCJRIHPRTDFWPZKKQPFTSJPVFOSJTQQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533939650 (559h17m16.405022685s) [noise=true , address=145.79.227.51:4410 , piece_id=5FX23RQJ5O65SKFTEW7BO2MP36VNI42RNYWUHE7PP4TZ4ZYSBYWQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534185308 (559h17m16.405268343s) [noise=true , address=145.79.227.36:4518 , piece_id=7TUGDNTYFP6AWE7ESKPDDU33ZBTUFOPTVMH4C5GRORO66DVTR65A, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534401649 (559h17m16.405484684s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534675090 (559h17m16.405758125s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534732289 (559h17m16.405815335s) [noise=true , address=145.79.227.37:4566 , piece_id=SASNC4CAPW2F5OUSZHSKRPOYUMMMSQ5G6KHZIW4NVFZM2RAEEZGQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535051748 (559h17m16.406134783s) [noise=true , address=145.79.227.11:4729 , piece_id=R25WH6YYDQ36YUIINIC3VNDEWESXMCO6K43YWE24EAQYCTELD6XA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535147068 (559h17m16.406230113s) [noise=true , address=45.91.116.13:4155 , piece_id=EDJCUZF5LE62KS3SDNRWMIY7UKXYP3JND7EXF3T5SX3OLKIHCLPA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535610190 (559h17m16.406693225s) [noise=true , address=145.79.227.38:4979 , piece_id=QYCUHBBHCQA2M4IEDWEA6TTGLU43FUKLQHSR2SQCMC4SHZNIJE3A, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.536199472 (559h17m16.407282517s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.536784084 (559h17m16.40786713s) [noise=true , address=45.91.116.5:4297 , piece_id=2MCIT75SXRVL5VE4CHCKRIDYWEDK54TA4ZTYF2ZQ4Z3FSDWIZ7MA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:18.203909763 (559h17m18.074992808s) [upload=490322]: Next returned nil segment Dec 28 22:46:18.203934080 (559h17m18.075017125s) [upload=490322]: Waiting for error group managing segments... Dec 28 22:46:18.533777259 (559h17m18.404860294s) [noise=true , address=145.79.227.16:4910 , piece_id=MDA4MGAKXBMGBRQSZ7WC4OS3MRQQ3VFJ3W3TYHXJB3SCB25XYNFA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.535160322 (559h17m18.406243367s) [noise=true , address=145.79.227.3:4150 , piece_id=KCXQM5B6E3QKFVFBIDIKLBNE4YMU4RZAUKTUWQJFHEBZNC26E2SA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.537196999 (559h17m18.408280044s) [noise=true , address=145.79.227.11:4729 , piece_id=R25WH6YYDQ36YUIINIC3VNDEWESXMCO6K43YWE24EAQYCTELD6XA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.548289885 (559h17m18.41937292s) [noise=true , address=145.79.227.51:4410 , piece_id=5FX23RQJ5O65SKFTEW7BO2MP36VNI42RNYWUHE7PP4TZ4ZYSBYWQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.548489425 (559h17m18.41957246s) [noise=true , address=145.79.227.42:4121 , piece_id=CALDXEHAPLIINBUJVB25J2Z6J6QHX3LVGVGNQ4YTTDZ2UMU2QSDA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.553480856 (559h17m18.424563891s) [noise=true , address=145.79.227.36:4518 , piece_id=7TUGDNTYFP6AWE7ESKPDDU33ZBTUFOPTVMH4C5GRORO66DVTR65A, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.554020704 (559h17m18.425103749s) [noise=true , address=145.79.227.37:4566 , piece_id=SASNC4CAPW2F5OUSZHSKRPOYUMMMSQ5G6KHZIW4NVFZM2RAEEZGQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.557268637 (559h17m18.428351672s) [noise=true , address=145.79.227.38:4979 , piece_id=QYCUHBBHCQA2M4IEDWEA6TTGLU43FUKLQHSR2SQCMC4SHZNIJE3A, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.561679774 (559h17m18.432762809s) [noise=true , address=145.79.227.27:4691 , piece_id=7F373BDGJHBKITEIYRQ7QJCJRIHPRTDFWPZKKQPFTSJPVFOSJTQQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.620428987 (559h17m18.491512022s) [noise=true , address=145.79.227.29:4941 , piece_id=ONW7MMHOMTQWMAAKKW43YXNB53KOZOQUA3TYDBLVT6HCVNKDWAOA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.734831199 (559h17m18.605914234s) [noise=true , address=45.91.116.5:4297 , piece_id=2MCIT75SXRVL5VE4CHCKRIDYWEDK54TA4ZTYF2ZQ4Z3FSDWIZ7MA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.764169667 (559h17m18.635252712s) [noise=true , address=45.91.116.13:4155 , piece_id=EDJCUZF5LE62KS3SDNRWMIY7UKXYP3JND7EXF3T5SX3OLKIHCLPA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.899179717 (559h17m18.770262752s) [noise=true , address=45.91.116.8:4047 , piece_id=OE32NXK6YJHX7TG36HPAHM7CQ7Z43YQMGVHADNZJCID7IA3PESHQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.899204645 (559h17m18.77028769s) [seg_pos={0 0}, upload=490322]: Segment reached optimal threshold of 13 pieces. Dec 28 22:46:18.899419233 (559h17m18.770502269s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899428721 (559h17m18.770511756s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899444932 (559h17m18.770527977s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899449651 (559h17m18.770532686s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899590659 (559h17m18.770673694s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899601921 (559h17m18.770684966s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899621057 (559h17m18.770704092s) [seg_pos={0 0}, upload=490322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:46:18.899821439 (559h17m18.770904474s) [upload=490322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:46:18.899827490 (559h17m18.770910535s) [upload=490322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:46:19.190656313 (559h17m19.061739348s) [upload=490322]: Done uploading segments... Dec 28 22:46:19.190698935 (559h17m19.06178197s) [upload=490322]: Upload finished. err: <nil> Duration: 6.043174266s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.X...................X.................] duration: 25.594889648s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:50:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 25.594912872s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X....................X.....X....X.X...X....X....] duration: 6.686060487s trace: https://jaeger.dev.storj.io/trace/10d2927d069b4ed1 FAIL (2025-12-28 22:49:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 22:46:16.281303410 (559h17m16.152386445s) [upload=490322]: Starting upload Dec 28 22:46:16.281379415 (559h17m16.15246246s) [upload=490322]: Done starting upload Dec 28 22:46:16.281492900 (559h17m16.152575945s) [upload=490322]: Uploading segments... Dec 28 22:46:16.281736144 (559h17m16.152819189s) [upload=490322]: Got next segment. Inline: false Dec 28 22:46:16.281827829 (559h17m16.152910874s) [upload=490322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:46:16.281833499 (559h17m16.152916534s) [upload=490322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:46:16.532054602 (559h17m16.403137637s) [seg_pos={0 0}, upload=490322]: Begin upload segment... Dec 28 22:46:16.532131298 (559h17m16.403214343s) [seg_pos={0 0}, upload=490322]: Done begin upload segment. Dec 28 22:46:16.532199558 (559h17m16.403282593s) [noise=true , address=45.91.116.8:4047 , piece_id=OE32NXK6YJHX7TG36HPAHM7CQ7Z43YQMGVHADNZJCID7IA3PESHQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.532247149 (559h17m16.403330184s) [noise=true , address=145.79.227.16:4910 , piece_id=MDA4MGAKXBMGBRQSZ7WC4OS3MRQQ3VFJ3W3TYHXJB3SCB25XYNFA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.532401703 (559h17m16.403484738s) [noise=true , address=145.79.227.29:4941 , piece_id=ONW7MMHOMTQWMAAKKW43YXNB53KOZOQUA3TYDBLVT6HCVNKDWAOA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533103579 (559h17m16.404186614s) [noise=true , address=145.79.227.42:4121 , piece_id=CALDXEHAPLIINBUJVB25J2Z6J6QHX3LVGVGNQ4YTTDZ2UMU2QSDA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533363634 (559h17m16.404446669s) [noise=true , address=145.79.227.3:4150 , piece_id=KCXQM5B6E3QKFVFBIDIKLBNE4YMU4RZAUKTUWQJFHEBZNC26E2SA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533498531 (559h17m16.404581576s) [noise=true , address=145.79.227.27:4691 , piece_id=7F373BDGJHBKITEIYRQ7QJCJRIHPRTDFWPZKKQPFTSJPVFOSJTQQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533939650 (559h17m16.405022685s) [noise=true , address=145.79.227.51:4410 , piece_id=5FX23RQJ5O65SKFTEW7BO2MP36VNI42RNYWUHE7PP4TZ4ZYSBYWQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534185308 (559h17m16.405268343s) [noise=true , address=145.79.227.36:4518 , piece_id=7TUGDNTYFP6AWE7ESKPDDU33ZBTUFOPTVMH4C5GRORO66DVTR65A, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534401649 (559h17m16.405484684s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534675090 (559h17m16.405758125s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534732289 (559h17m16.405815335s) [noise=true , address=145.79.227.37:4566 , piece_id=SASNC4CAPW2F5OUSZHSKRPOYUMMMSQ5G6KHZIW4NVFZM2RAEEZGQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535051748 (559h17m16.406134783s) [noise=true , address=145.79.227.11:4729 , piece_id=R25WH6YYDQ36YUIINIC3VNDEWESXMCO6K43YWE24EAQYCTELD6XA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535147068 (559h17m16.406230113s) [noise=true , address=45.91.116.13:4155 , piece_id=EDJCUZF5LE62KS3SDNRWMIY7UKXYP3JND7EXF3T5SX3OLKIHCLPA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535610190 (559h17m16.406693225s) [noise=true , address=145.79.227.38:4979 , piece_id=QYCUHBBHCQA2M4IEDWEA6TTGLU43FUKLQHSR2SQCMC4SHZNIJE3A, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.536199472 (559h17m16.407282517s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.536784084 (559h17m16.40786713s) [noise=true , address=45.91.116.5:4297 , piece_id=2MCIT75SXRVL5VE4CHCKRIDYWEDK54TA4ZTYF2ZQ4Z3FSDWIZ7MA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:18.203909763 (559h17m18.074992808s) [upload=490322]: Next returned nil segment Dec 28 22:46:18.203934080 (559h17m18.075017125s) [upload=490322]: Waiting for error group managing segments... Dec 28 22:46:18.533777259 (559h17m18.404860294s) [noise=true , address=145.79.227.16:4910 , piece_id=MDA4MGAKXBMGBRQSZ7WC4OS3MRQQ3VFJ3W3TYHXJB3SCB25XYNFA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.535160322 (559h17m18.406243367s) [noise=true , address=145.79.227.3:4150 , piece_id=KCXQM5B6E3QKFVFBIDIKLBNE4YMU4RZAUKTUWQJFHEBZNC26E2SA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.537196999 (559h17m18.408280044s) [noise=true , address=145.79.227.11:4729 , piece_id=R25WH6YYDQ36YUIINIC3VNDEWESXMCO6K43YWE24EAQYCTELD6XA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.548289885 (559h17m18.41937292s) [noise=true , address=145.79.227.51:4410 , piece_id=5FX23RQJ5O65SKFTEW7BO2MP36VNI42RNYWUHE7PP4TZ4ZYSBYWQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.548489425 (559h17m18.41957246s) [noise=true , address=145.79.227.42:4121 , piece_id=CALDXEHAPLIINBUJVB25J2Z6J6QHX3LVGVGNQ4YTTDZ2UMU2QSDA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.553480856 (559h17m18.424563891s) [noise=true , address=145.79.227.36:4518 , piece_id=7TUGDNTYFP6AWE7ESKPDDU33ZBTUFOPTVMH4C5GRORO66DVTR65A, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.554020704 (559h17m18.425103749s) [noise=true , address=145.79.227.37:4566 , piece_id=SASNC4CAPW2F5OUSZHSKRPOYUMMMSQ5G6KHZIW4NVFZM2RAEEZGQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.557268637 (559h17m18.428351672s) [noise=true , address=145.79.227.38:4979 , piece_id=QYCUHBBHCQA2M4IEDWEA6TTGLU43FUKLQHSR2SQCMC4SHZNIJE3A, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.561679774 (559h17m18.432762809s) [noise=true , address=145.79.227.27:4691 , piece_id=7F373BDGJHBKITEIYRQ7QJCJRIHPRTDFWPZKKQPFTSJPVFOSJTQQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.620428987 (559h17m18.491512022s) [noise=true , address=145.79.227.29:4941 , piece_id=ONW7MMHOMTQWMAAKKW43YXNB53KOZOQUA3TYDBLVT6HCVNKDWAOA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.734831199 (559h17m18.605914234s) [noise=true , address=45.91.116.5:4297 , piece_id=2MCIT75SXRVL5VE4CHCKRIDYWEDK54TA4ZTYF2ZQ4Z3FSDWIZ7MA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.764169667 (559h17m18.635252712s) [noise=true , address=45.91.116.13:4155 , piece_id=EDJCUZF5LE62KS3SDNRWMIY7UKXYP3JND7EXF3T5SX3OLKIHCLPA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.899179717 (559h17m18.770262752s) [noise=true , address=45.91.116.8:4047 , piece_id=OE32NXK6YJHX7TG36HPAHM7CQ7Z43YQMGVHADNZJCID7IA3PESHQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.899204645 (559h17m18.77028769s) [seg_pos={0 0}, upload=490322]: Segment reached optimal threshold of 13 pieces. Dec 28 22:46:18.899419233 (559h17m18.770502269s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899428721 (559h17m18.770511756s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899444932 (559h17m18.770527977s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899449651 (559h17m18.770532686s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899590659 (559h17m18.770673694s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899601921 (559h17m18.770684966s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899621057 (559h17m18.770704092s) [seg_pos={0 0}, upload=490322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:46:18.899821439 (559h17m18.770904474s) [upload=490322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:46:18.899827490 (559h17m18.770910535s) [upload=490322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:46:19.190656313 (559h17m19.061739348s) [upload=490322]: Done uploading segments... Dec 28 22:46:19.190698935 (559h17m19.06178197s) [upload=490322]: Upload finished. err: <nil> Duration: 6.686085816s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.X...................X................] duration: 17.325072838s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:49:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 17.325100139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............X....................X.....X....X.X...X....X...] duration: 5.990554884s trace: https://jaeger.dev.storj.io/trace/10d2927d069b4ed1 FAIL (2025-12-28 22:49:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 22:46:16.281303410 (559h17m16.152386445s) [upload=490322]: Starting upload Dec 28 22:46:16.281379415 (559h17m16.15246246s) [upload=490322]: Done starting upload Dec 28 22:46:16.281492900 (559h17m16.152575945s) [upload=490322]: Uploading segments... Dec 28 22:46:16.281736144 (559h17m16.152819189s) [upload=490322]: Got next segment. Inline: false Dec 28 22:46:16.281827829 (559h17m16.152910874s) [upload=490322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:46:16.281833499 (559h17m16.152916534s) [upload=490322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:46:16.532054602 (559h17m16.403137637s) [seg_pos={0 0}, upload=490322]: Begin upload segment... Dec 28 22:46:16.532131298 (559h17m16.403214343s) [seg_pos={0 0}, upload=490322]: Done begin upload segment. Dec 28 22:46:16.532199558 (559h17m16.403282593s) [noise=true , address=45.91.116.8:4047 , piece_id=OE32NXK6YJHX7TG36HPAHM7CQ7Z43YQMGVHADNZJCID7IA3PESHQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.532247149 (559h17m16.403330184s) [noise=true , address=145.79.227.16:4910 , piece_id=MDA4MGAKXBMGBRQSZ7WC4OS3MRQQ3VFJ3W3TYHXJB3SCB25XYNFA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.532401703 (559h17m16.403484738s) [noise=true , address=145.79.227.29:4941 , piece_id=ONW7MMHOMTQWMAAKKW43YXNB53KOZOQUA3TYDBLVT6HCVNKDWAOA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533103579 (559h17m16.404186614s) [noise=true , address=145.79.227.42:4121 , piece_id=CALDXEHAPLIINBUJVB25J2Z6J6QHX3LVGVGNQ4YTTDZ2UMU2QSDA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533363634 (559h17m16.404446669s) [noise=true , address=145.79.227.3:4150 , piece_id=KCXQM5B6E3QKFVFBIDIKLBNE4YMU4RZAUKTUWQJFHEBZNC26E2SA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533498531 (559h17m16.404581576s) [noise=true , address=145.79.227.27:4691 , piece_id=7F373BDGJHBKITEIYRQ7QJCJRIHPRTDFWPZKKQPFTSJPVFOSJTQQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533939650 (559h17m16.405022685s) [noise=true , address=145.79.227.51:4410 , piece_id=5FX23RQJ5O65SKFTEW7BO2MP36VNI42RNYWUHE7PP4TZ4ZYSBYWQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534185308 (559h17m16.405268343s) [noise=true , address=145.79.227.36:4518 , piece_id=7TUGDNTYFP6AWE7ESKPDDU33ZBTUFOPTVMH4C5GRORO66DVTR65A, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534401649 (559h17m16.405484684s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534675090 (559h17m16.405758125s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534732289 (559h17m16.405815335s) [noise=true , address=145.79.227.37:4566 , piece_id=SASNC4CAPW2F5OUSZHSKRPOYUMMMSQ5G6KHZIW4NVFZM2RAEEZGQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535051748 (559h17m16.406134783s) [noise=true , address=145.79.227.11:4729 , piece_id=R25WH6YYDQ36YUIINIC3VNDEWESXMCO6K43YWE24EAQYCTELD6XA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535147068 (559h17m16.406230113s) [noise=true , address=45.91.116.13:4155 , piece_id=EDJCUZF5LE62KS3SDNRWMIY7UKXYP3JND7EXF3T5SX3OLKIHCLPA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535610190 (559h17m16.406693225s) [noise=true , address=145.79.227.38:4979 , piece_id=QYCUHBBHCQA2M4IEDWEA6TTGLU43FUKLQHSR2SQCMC4SHZNIJE3A, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.536199472 (559h17m16.407282517s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.536784084 (559h17m16.40786713s) [noise=true , address=45.91.116.5:4297 , piece_id=2MCIT75SXRVL5VE4CHCKRIDYWEDK54TA4ZTYF2ZQ4Z3FSDWIZ7MA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:18.203909763 (559h17m18.074992808s) [upload=490322]: Next returned nil segment Dec 28 22:46:18.203934080 (559h17m18.075017125s) [upload=490322]: Waiting for error group managing segments... Dec 28 22:46:18.533777259 (559h17m18.404860294s) [noise=true , address=145.79.227.16:4910 , piece_id=MDA4MGAKXBMGBRQSZ7WC4OS3MRQQ3VFJ3W3TYHXJB3SCB25XYNFA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.535160322 (559h17m18.406243367s) [noise=true , address=145.79.227.3:4150 , piece_id=KCXQM5B6E3QKFVFBIDIKLBNE4YMU4RZAUKTUWQJFHEBZNC26E2SA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.537196999 (559h17m18.408280044s) [noise=true , address=145.79.227.11:4729 , piece_id=R25WH6YYDQ36YUIINIC3VNDEWESXMCO6K43YWE24EAQYCTELD6XA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.548289885 (559h17m18.41937292s) [noise=true , address=145.79.227.51:4410 , piece_id=5FX23RQJ5O65SKFTEW7BO2MP36VNI42RNYWUHE7PP4TZ4ZYSBYWQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.548489425 (559h17m18.41957246s) [noise=true , address=145.79.227.42:4121 , piece_id=CALDXEHAPLIINBUJVB25J2Z6J6QHX3LVGVGNQ4YTTDZ2UMU2QSDA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.553480856 (559h17m18.424563891s) [noise=true , address=145.79.227.36:4518 , piece_id=7TUGDNTYFP6AWE7ESKPDDU33ZBTUFOPTVMH4C5GRORO66DVTR65A, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.554020704 (559h17m18.425103749s) [noise=true , address=145.79.227.37:4566 , piece_id=SASNC4CAPW2F5OUSZHSKRPOYUMMMSQ5G6KHZIW4NVFZM2RAEEZGQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.557268637 (559h17m18.428351672s) [noise=true , address=145.79.227.38:4979 , piece_id=QYCUHBBHCQA2M4IEDWEA6TTGLU43FUKLQHSR2SQCMC4SHZNIJE3A, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.561679774 (559h17m18.432762809s) [noise=true , address=145.79.227.27:4691 , piece_id=7F373BDGJHBKITEIYRQ7QJCJRIHPRTDFWPZKKQPFTSJPVFOSJTQQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.620428987 (559h17m18.491512022s) [noise=true , address=145.79.227.29:4941 , piece_id=ONW7MMHOMTQWMAAKKW43YXNB53KOZOQUA3TYDBLVT6HCVNKDWAOA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.734831199 (559h17m18.605914234s) [noise=true , address=45.91.116.5:4297 , piece_id=2MCIT75SXRVL5VE4CHCKRIDYWEDK54TA4ZTYF2ZQ4Z3FSDWIZ7MA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.764169667 (559h17m18.635252712s) [noise=true , address=45.91.116.13:4155 , piece_id=EDJCUZF5LE62KS3SDNRWMIY7UKXYP3JND7EXF3T5SX3OLKIHCLPA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.899179717 (559h17m18.770262752s) [noise=true , address=45.91.116.8:4047 , piece_id=OE32NXK6YJHX7TG36HPAHM7CQ7Z43YQMGVHADNZJCID7IA3PESHQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.899204645 (559h17m18.77028769s) [seg_pos={0 0}, upload=490322]: Segment reached optimal threshold of 13 pieces. Dec 28 22:46:18.899419233 (559h17m18.770502269s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899428721 (559h17m18.770511756s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899444932 (559h17m18.770527977s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899449651 (559h17m18.770532686s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899590659 (559h17m18.770673694s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899601921 (559h17m18.770684966s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899621057 (559h17m18.770704092s) [seg_pos={0 0}, upload=490322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:46:18.899821439 (559h17m18.770904474s) [upload=490322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:46:18.899827490 (559h17m18.770910535s) [upload=490322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:46:19.190656313 (559h17m19.061739348s) [upload=490322]: Done uploading segments... Dec 28 22:46:19.190698935 (559h17m19.06178197s) [upload=490322]: Upload finished. err: <nil> Duration: 5.990579932s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.X...................X...............] duration: 17.560300252s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:48:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 17.560318726s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............X....................X.....X....X.X...X....X..] duration: 6.57922111s trace: https://jaeger.dev.storj.io/trace/10d2927d069b4ed1 FAIL (2025-12-28 22:48:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 22:46:16.281303410 (559h17m16.152386445s) [upload=490322]: Starting upload Dec 28 22:46:16.281379415 (559h17m16.15246246s) [upload=490322]: Done starting upload Dec 28 22:46:16.281492900 (559h17m16.152575945s) [upload=490322]: Uploading segments... Dec 28 22:46:16.281736144 (559h17m16.152819189s) [upload=490322]: Got next segment. Inline: false Dec 28 22:46:16.281827829 (559h17m16.152910874s) [upload=490322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:46:16.281833499 (559h17m16.152916534s) [upload=490322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:46:16.532054602 (559h17m16.403137637s) [seg_pos={0 0}, upload=490322]: Begin upload segment... Dec 28 22:46:16.532131298 (559h17m16.403214343s) [seg_pos={0 0}, upload=490322]: Done begin upload segment. Dec 28 22:46:16.532199558 (559h17m16.403282593s) [noise=true , address=45.91.116.8:4047 , piece_id=OE32NXK6YJHX7TG36HPAHM7CQ7Z43YQMGVHADNZJCID7IA3PESHQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.532247149 (559h17m16.403330184s) [noise=true , address=145.79.227.16:4910 , piece_id=MDA4MGAKXBMGBRQSZ7WC4OS3MRQQ3VFJ3W3TYHXJB3SCB25XYNFA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.532401703 (559h17m16.403484738s) [noise=true , address=145.79.227.29:4941 , piece_id=ONW7MMHOMTQWMAAKKW43YXNB53KOZOQUA3TYDBLVT6HCVNKDWAOA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533103579 (559h17m16.404186614s) [noise=true , address=145.79.227.42:4121 , piece_id=CALDXEHAPLIINBUJVB25J2Z6J6QHX3LVGVGNQ4YTTDZ2UMU2QSDA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533363634 (559h17m16.404446669s) [noise=true , address=145.79.227.3:4150 , piece_id=KCXQM5B6E3QKFVFBIDIKLBNE4YMU4RZAUKTUWQJFHEBZNC26E2SA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533498531 (559h17m16.404581576s) [noise=true , address=145.79.227.27:4691 , piece_id=7F373BDGJHBKITEIYRQ7QJCJRIHPRTDFWPZKKQPFTSJPVFOSJTQQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533939650 (559h17m16.405022685s) [noise=true , address=145.79.227.51:4410 , piece_id=5FX23RQJ5O65SKFTEW7BO2MP36VNI42RNYWUHE7PP4TZ4ZYSBYWQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534185308 (559h17m16.405268343s) [noise=true , address=145.79.227.36:4518 , piece_id=7TUGDNTYFP6AWE7ESKPDDU33ZBTUFOPTVMH4C5GRORO66DVTR65A, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534401649 (559h17m16.405484684s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534675090 (559h17m16.405758125s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534732289 (559h17m16.405815335s) [noise=true , address=145.79.227.37:4566 , piece_id=SASNC4CAPW2F5OUSZHSKRPOYUMMMSQ5G6KHZIW4NVFZM2RAEEZGQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535051748 (559h17m16.406134783s) [noise=true , address=145.79.227.11:4729 , piece_id=R25WH6YYDQ36YUIINIC3VNDEWESXMCO6K43YWE24EAQYCTELD6XA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535147068 (559h17m16.406230113s) [noise=true , address=45.91.116.13:4155 , piece_id=EDJCUZF5LE62KS3SDNRWMIY7UKXYP3JND7EXF3T5SX3OLKIHCLPA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535610190 (559h17m16.406693225s) [noise=true , address=145.79.227.38:4979 , piece_id=QYCUHBBHCQA2M4IEDWEA6TTGLU43FUKLQHSR2SQCMC4SHZNIJE3A, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.536199472 (559h17m16.407282517s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.536784084 (559h17m16.40786713s) [noise=true , address=45.91.116.5:4297 , piece_id=2MCIT75SXRVL5VE4CHCKRIDYWEDK54TA4ZTYF2ZQ4Z3FSDWIZ7MA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:18.203909763 (559h17m18.074992808s) [upload=490322]: Next returned nil segment Dec 28 22:46:18.203934080 (559h17m18.075017125s) [upload=490322]: Waiting for error group managing segments... Dec 28 22:46:18.533777259 (559h17m18.404860294s) [noise=true , address=145.79.227.16:4910 , piece_id=MDA4MGAKXBMGBRQSZ7WC4OS3MRQQ3VFJ3W3TYHXJB3SCB25XYNFA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.535160322 (559h17m18.406243367s) [noise=true , address=145.79.227.3:4150 , piece_id=KCXQM5B6E3QKFVFBIDIKLBNE4YMU4RZAUKTUWQJFHEBZNC26E2SA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.537196999 (559h17m18.408280044s) [noise=true , address=145.79.227.11:4729 , piece_id=R25WH6YYDQ36YUIINIC3VNDEWESXMCO6K43YWE24EAQYCTELD6XA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.548289885 (559h17m18.41937292s) [noise=true , address=145.79.227.51:4410 , piece_id=5FX23RQJ5O65SKFTEW7BO2MP36VNI42RNYWUHE7PP4TZ4ZYSBYWQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.548489425 (559h17m18.41957246s) [noise=true , address=145.79.227.42:4121 , piece_id=CALDXEHAPLIINBUJVB25J2Z6J6QHX3LVGVGNQ4YTTDZ2UMU2QSDA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.553480856 (559h17m18.424563891s) [noise=true , address=145.79.227.36:4518 , piece_id=7TUGDNTYFP6AWE7ESKPDDU33ZBTUFOPTVMH4C5GRORO66DVTR65A, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.554020704 (559h17m18.425103749s) [noise=true , address=145.79.227.37:4566 , piece_id=SASNC4CAPW2F5OUSZHSKRPOYUMMMSQ5G6KHZIW4NVFZM2RAEEZGQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.557268637 (559h17m18.428351672s) [noise=true , address=145.79.227.38:4979 , piece_id=QYCUHBBHCQA2M4IEDWEA6TTGLU43FUKLQHSR2SQCMC4SHZNIJE3A, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.561679774 (559h17m18.432762809s) [noise=true , address=145.79.227.27:4691 , piece_id=7F373BDGJHBKITEIYRQ7QJCJRIHPRTDFWPZKKQPFTSJPVFOSJTQQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.620428987 (559h17m18.491512022s) [noise=true , address=145.79.227.29:4941 , piece_id=ONW7MMHOMTQWMAAKKW43YXNB53KOZOQUA3TYDBLVT6HCVNKDWAOA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.734831199 (559h17m18.605914234s) [noise=true , address=45.91.116.5:4297 , piece_id=2MCIT75SXRVL5VE4CHCKRIDYWEDK54TA4ZTYF2ZQ4Z3FSDWIZ7MA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.764169667 (559h17m18.635252712s) [noise=true , address=45.91.116.13:4155 , piece_id=EDJCUZF5LE62KS3SDNRWMIY7UKXYP3JND7EXF3T5SX3OLKIHCLPA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.899179717 (559h17m18.770262752s) [noise=true , address=45.91.116.8:4047 , piece_id=OE32NXK6YJHX7TG36HPAHM7CQ7Z43YQMGVHADNZJCID7IA3PESHQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.899204645 (559h17m18.77028769s) [seg_pos={0 0}, upload=490322]: Segment reached optimal threshold of 13 pieces. Dec 28 22:46:18.899419233 (559h17m18.770502269s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899428721 (559h17m18.770511756s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899444932 (559h17m18.770527977s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899449651 (559h17m18.770532686s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899590659 (559h17m18.770673694s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899601921 (559h17m18.770684966s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899621057 (559h17m18.770704092s) [seg_pos={0 0}, upload=490322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:46:18.899821439 (559h17m18.770904474s) [upload=490322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:46:18.899827490 (559h17m18.770910535s) [upload=490322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:46:19.190656313 (559h17m19.061739348s) [upload=490322]: Done uploading segments... Dec 28 22:46:19.190698935 (559h17m19.06178197s) [upload=490322]: Upload finished. err: <nil> Duration: 6.579245586s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.X...................X..............] duration: 39.373078529s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:47:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 39.373103677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............X....................X.....X....X.X...X....X.] duration: 6.185538367s trace: https://jaeger.dev.storj.io/trace/10d2927d069b4ed1 FAIL (2025-12-28 22:46:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 22:46:16.281303410 (559h17m16.152386445s) [upload=490322]: Starting upload Dec 28 22:46:16.281379415 (559h17m16.15246246s) [upload=490322]: Done starting upload Dec 28 22:46:16.281492900 (559h17m16.152575945s) [upload=490322]: Uploading segments... Dec 28 22:46:16.281736144 (559h17m16.152819189s) [upload=490322]: Got next segment. Inline: false Dec 28 22:46:16.281827829 (559h17m16.152910874s) [upload=490322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:46:16.281833499 (559h17m16.152916534s) [upload=490322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:46:16.532054602 (559h17m16.403137637s) [seg_pos={0 0}, upload=490322]: Begin upload segment... Dec 28 22:46:16.532131298 (559h17m16.403214343s) [seg_pos={0 0}, upload=490322]: Done begin upload segment. Dec 28 22:46:16.532199558 (559h17m16.403282593s) [noise=true , address=45.91.116.8:4047 , piece_id=OE32NXK6YJHX7TG36HPAHM7CQ7Z43YQMGVHADNZJCID7IA3PESHQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.532247149 (559h17m16.403330184s) [noise=true , address=145.79.227.16:4910 , piece_id=MDA4MGAKXBMGBRQSZ7WC4OS3MRQQ3VFJ3W3TYHXJB3SCB25XYNFA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.532401703 (559h17m16.403484738s) [noise=true , address=145.79.227.29:4941 , piece_id=ONW7MMHOMTQWMAAKKW43YXNB53KOZOQUA3TYDBLVT6HCVNKDWAOA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533103579 (559h17m16.404186614s) [noise=true , address=145.79.227.42:4121 , piece_id=CALDXEHAPLIINBUJVB25J2Z6J6QHX3LVGVGNQ4YTTDZ2UMU2QSDA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533363634 (559h17m16.404446669s) [noise=true , address=145.79.227.3:4150 , piece_id=KCXQM5B6E3QKFVFBIDIKLBNE4YMU4RZAUKTUWQJFHEBZNC26E2SA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533498531 (559h17m16.404581576s) [noise=true , address=145.79.227.27:4691 , piece_id=7F373BDGJHBKITEIYRQ7QJCJRIHPRTDFWPZKKQPFTSJPVFOSJTQQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533939650 (559h17m16.405022685s) [noise=true , address=145.79.227.51:4410 , piece_id=5FX23RQJ5O65SKFTEW7BO2MP36VNI42RNYWUHE7PP4TZ4ZYSBYWQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534185308 (559h17m16.405268343s) [noise=true , address=145.79.227.36:4518 , piece_id=7TUGDNTYFP6AWE7ESKPDDU33ZBTUFOPTVMH4C5GRORO66DVTR65A, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534401649 (559h17m16.405484684s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534675090 (559h17m16.405758125s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534732289 (559h17m16.405815335s) [noise=true , address=145.79.227.37:4566 , piece_id=SASNC4CAPW2F5OUSZHSKRPOYUMMMSQ5G6KHZIW4NVFZM2RAEEZGQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535051748 (559h17m16.406134783s) [noise=true , address=145.79.227.11:4729 , piece_id=R25WH6YYDQ36YUIINIC3VNDEWESXMCO6K43YWE24EAQYCTELD6XA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535147068 (559h17m16.406230113s) [noise=true , address=45.91.116.13:4155 , piece_id=EDJCUZF5LE62KS3SDNRWMIY7UKXYP3JND7EXF3T5SX3OLKIHCLPA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535610190 (559h17m16.406693225s) [noise=true , address=145.79.227.38:4979 , piece_id=QYCUHBBHCQA2M4IEDWEA6TTGLU43FUKLQHSR2SQCMC4SHZNIJE3A, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.536199472 (559h17m16.407282517s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.536784084 (559h17m16.40786713s) [noise=true , address=45.91.116.5:4297 , piece_id=2MCIT75SXRVL5VE4CHCKRIDYWEDK54TA4ZTYF2ZQ4Z3FSDWIZ7MA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:18.203909763 (559h17m18.074992808s) [upload=490322]: Next returned nil segment Dec 28 22:46:18.203934080 (559h17m18.075017125s) [upload=490322]: Waiting for error group managing segments... Dec 28 22:46:18.533777259 (559h17m18.404860294s) [noise=true , address=145.79.227.16:4910 , piece_id=MDA4MGAKXBMGBRQSZ7WC4OS3MRQQ3VFJ3W3TYHXJB3SCB25XYNFA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.535160322 (559h17m18.406243367s) [noise=true , address=145.79.227.3:4150 , piece_id=KCXQM5B6E3QKFVFBIDIKLBNE4YMU4RZAUKTUWQJFHEBZNC26E2SA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.537196999 (559h17m18.408280044s) [noise=true , address=145.79.227.11:4729 , piece_id=R25WH6YYDQ36YUIINIC3VNDEWESXMCO6K43YWE24EAQYCTELD6XA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.548289885 (559h17m18.41937292s) [noise=true , address=145.79.227.51:4410 , piece_id=5FX23RQJ5O65SKFTEW7BO2MP36VNI42RNYWUHE7PP4TZ4ZYSBYWQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.548489425 (559h17m18.41957246s) [noise=true , address=145.79.227.42:4121 , piece_id=CALDXEHAPLIINBUJVB25J2Z6J6QHX3LVGVGNQ4YTTDZ2UMU2QSDA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.553480856 (559h17m18.424563891s) [noise=true , address=145.79.227.36:4518 , piece_id=7TUGDNTYFP6AWE7ESKPDDU33ZBTUFOPTVMH4C5GRORO66DVTR65A, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.554020704 (559h17m18.425103749s) [noise=true , address=145.79.227.37:4566 , piece_id=SASNC4CAPW2F5OUSZHSKRPOYUMMMSQ5G6KHZIW4NVFZM2RAEEZGQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.557268637 (559h17m18.428351672s) [noise=true , address=145.79.227.38:4979 , piece_id=QYCUHBBHCQA2M4IEDWEA6TTGLU43FUKLQHSR2SQCMC4SHZNIJE3A, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.561679774 (559h17m18.432762809s) [noise=true , address=145.79.227.27:4691 , piece_id=7F373BDGJHBKITEIYRQ7QJCJRIHPRTDFWPZKKQPFTSJPVFOSJTQQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.620428987 (559h17m18.491512022s) [noise=true , address=145.79.227.29:4941 , piece_id=ONW7MMHOMTQWMAAKKW43YXNB53KOZOQUA3TYDBLVT6HCVNKDWAOA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.734831199 (559h17m18.605914234s) [noise=true , address=45.91.116.5:4297 , piece_id=2MCIT75SXRVL5VE4CHCKRIDYWEDK54TA4ZTYF2ZQ4Z3FSDWIZ7MA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.764169667 (559h17m18.635252712s) [noise=true , address=45.91.116.13:4155 , piece_id=EDJCUZF5LE62KS3SDNRWMIY7UKXYP3JND7EXF3T5SX3OLKIHCLPA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.899179717 (559h17m18.770262752s) [noise=true , address=45.91.116.8:4047 , piece_id=OE32NXK6YJHX7TG36HPAHM7CQ7Z43YQMGVHADNZJCID7IA3PESHQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.899204645 (559h17m18.77028769s) [seg_pos={0 0}, upload=490322]: Segment reached optimal threshold of 13 pieces. Dec 28 22:46:18.899419233 (559h17m18.770502269s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899428721 (559h17m18.770511756s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899444932 (559h17m18.770527977s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899449651 (559h17m18.770532686s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899590659 (559h17m18.770673694s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899601921 (559h17m18.770684966s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899621057 (559h17m18.770704092s) [seg_pos={0 0}, upload=490322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:46:18.899821439 (559h17m18.770904474s) [upload=490322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:46:18.899827490 (559h17m18.770910535s) [upload=490322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:46:19.190656313 (559h17m19.061739348s) [upload=490322]: Done uploading segments... Dec 28 22:46:19.190698935 (559h17m19.06178197s) [upload=490322]: Upload finished. err: <nil> Duration: 6.185567393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [................X....................X.....X....X.X...X....X] duration: 30.002737834s trace: https://jaeger.dev.storj.io/trace/10d2927d069b4ed1 FAIL (2025-12-28 22:46:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 22:46:16.281303410 (559h17m16.152386445s) [upload=490322]: Starting upload Dec 28 22:46:16.281379415 (559h17m16.15246246s) [upload=490322]: Done starting upload Dec 28 22:46:16.281492900 (559h17m16.152575945s) [upload=490322]: Uploading segments... Dec 28 22:46:16.281736144 (559h17m16.152819189s) [upload=490322]: Got next segment. Inline: false Dec 28 22:46:16.281827829 (559h17m16.152910874s) [upload=490322]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:46:16.281833499 (559h17m16.152916534s) [upload=490322]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:46:16.532054602 (559h17m16.403137637s) [seg_pos={0 0}, upload=490322]: Begin upload segment... Dec 28 22:46:16.532131298 (559h17m16.403214343s) [seg_pos={0 0}, upload=490322]: Done begin upload segment. Dec 28 22:46:16.532199558 (559h17m16.403282593s) [noise=true , address=45.91.116.8:4047 , piece_id=OE32NXK6YJHX7TG36HPAHM7CQ7Z43YQMGVHADNZJCID7IA3PESHQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.532247149 (559h17m16.403330184s) [noise=true , address=145.79.227.16:4910 , piece_id=MDA4MGAKXBMGBRQSZ7WC4OS3MRQQ3VFJ3W3TYHXJB3SCB25XYNFA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.532401703 (559h17m16.403484738s) [noise=true , address=145.79.227.29:4941 , piece_id=ONW7MMHOMTQWMAAKKW43YXNB53KOZOQUA3TYDBLVT6HCVNKDWAOA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533103579 (559h17m16.404186614s) [noise=true , address=145.79.227.42:4121 , piece_id=CALDXEHAPLIINBUJVB25J2Z6J6QHX3LVGVGNQ4YTTDZ2UMU2QSDA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533363634 (559h17m16.404446669s) [noise=true , address=145.79.227.3:4150 , piece_id=KCXQM5B6E3QKFVFBIDIKLBNE4YMU4RZAUKTUWQJFHEBZNC26E2SA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533498531 (559h17m16.404581576s) [noise=true , address=145.79.227.27:4691 , piece_id=7F373BDGJHBKITEIYRQ7QJCJRIHPRTDFWPZKKQPFTSJPVFOSJTQQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.533939650 (559h17m16.405022685s) [noise=true , address=145.79.227.51:4410 , piece_id=5FX23RQJ5O65SKFTEW7BO2MP36VNI42RNYWUHE7PP4TZ4ZYSBYWQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534185308 (559h17m16.405268343s) [noise=true , address=145.79.227.36:4518 , piece_id=7TUGDNTYFP6AWE7ESKPDDU33ZBTUFOPTVMH4C5GRORO66DVTR65A, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534401649 (559h17m16.405484684s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534675090 (559h17m16.405758125s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.534732289 (559h17m16.405815335s) [noise=true , address=145.79.227.37:4566 , piece_id=SASNC4CAPW2F5OUSZHSKRPOYUMMMSQ5G6KHZIW4NVFZM2RAEEZGQ, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535051748 (559h17m16.406134783s) [noise=true , address=145.79.227.11:4729 , piece_id=R25WH6YYDQ36YUIINIC3VNDEWESXMCO6K43YWE24EAQYCTELD6XA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535147068 (559h17m16.406230113s) [noise=true , address=45.91.116.13:4155 , piece_id=EDJCUZF5LE62KS3SDNRWMIY7UKXYP3JND7EXF3T5SX3OLKIHCLPA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.535610190 (559h17m16.406693225s) [noise=true , address=145.79.227.38:4979 , piece_id=QYCUHBBHCQA2M4IEDWEA6TTGLU43FUKLQHSR2SQCMC4SHZNIJE3A, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.536199472 (559h17m16.407282517s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:16.536784084 (559h17m16.40786713s) [noise=true , address=45.91.116.5:4297 , piece_id=2MCIT75SXRVL5VE4CHCKRIDYWEDK54TA4ZTYF2ZQ4Z3FSDWIZ7MA, seg_pos={0 0}, upload=490322]: Uploading piece... Dec 28 22:46:18.203909763 (559h17m18.074992808s) [upload=490322]: Next returned nil segment Dec 28 22:46:18.203934080 (559h17m18.075017125s) [upload=490322]: Waiting for error group managing segments... Dec 28 22:46:18.533777259 (559h17m18.404860294s) [noise=true , address=145.79.227.16:4910 , piece_id=MDA4MGAKXBMGBRQSZ7WC4OS3MRQQ3VFJ3W3TYHXJB3SCB25XYNFA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.535160322 (559h17m18.406243367s) [noise=true , address=145.79.227.3:4150 , piece_id=KCXQM5B6E3QKFVFBIDIKLBNE4YMU4RZAUKTUWQJFHEBZNC26E2SA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.537196999 (559h17m18.408280044s) [noise=true , address=145.79.227.11:4729 , piece_id=R25WH6YYDQ36YUIINIC3VNDEWESXMCO6K43YWE24EAQYCTELD6XA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.548289885 (559h17m18.41937292s) [noise=true , address=145.79.227.51:4410 , piece_id=5FX23RQJ5O65SKFTEW7BO2MP36VNI42RNYWUHE7PP4TZ4ZYSBYWQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.548489425 (559h17m18.41957246s) [noise=true , address=145.79.227.42:4121 , piece_id=CALDXEHAPLIINBUJVB25J2Z6J6QHX3LVGVGNQ4YTTDZ2UMU2QSDA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.553480856 (559h17m18.424563891s) [noise=true , address=145.79.227.36:4518 , piece_id=7TUGDNTYFP6AWE7ESKPDDU33ZBTUFOPTVMH4C5GRORO66DVTR65A, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.554020704 (559h17m18.425103749s) [noise=true , address=145.79.227.37:4566 , piece_id=SASNC4CAPW2F5OUSZHSKRPOYUMMMSQ5G6KHZIW4NVFZM2RAEEZGQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.557268637 (559h17m18.428351672s) [noise=true , address=145.79.227.38:4979 , piece_id=QYCUHBBHCQA2M4IEDWEA6TTGLU43FUKLQHSR2SQCMC4SHZNIJE3A, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.561679774 (559h17m18.432762809s) [noise=true , address=145.79.227.27:4691 , piece_id=7F373BDGJHBKITEIYRQ7QJCJRIHPRTDFWPZKKQPFTSJPVFOSJTQQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.620428987 (559h17m18.491512022s) [noise=true , address=145.79.227.29:4941 , piece_id=ONW7MMHOMTQWMAAKKW43YXNB53KOZOQUA3TYDBLVT6HCVNKDWAOA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.734831199 (559h17m18.605914234s) [noise=true , address=45.91.116.5:4297 , piece_id=2MCIT75SXRVL5VE4CHCKRIDYWEDK54TA4ZTYF2ZQ4Z3FSDWIZ7MA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.764169667 (559h17m18.635252712s) [noise=true , address=45.91.116.13:4155 , piece_id=EDJCUZF5LE62KS3SDNRWMIY7UKXYP3JND7EXF3T5SX3OLKIHCLPA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.899179717 (559h17m18.770262752s) [noise=true , address=45.91.116.8:4047 , piece_id=OE32NXK6YJHX7TG36HPAHM7CQ7Z43YQMGVHADNZJCID7IA3PESHQ, seg_pos={0 0}, upload=490322]: Done uploading piece. err: <nil> Dec 28 22:46:18.899204645 (559h17m18.77028769s) [seg_pos={0 0}, upload=490322]: Segment reached optimal threshold of 13 pieces. Dec 28 22:46:18.899419233 (559h17m18.770502269s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899428721 (559h17m18.770511756s) [noise=true , address=145.79.227.43:4159 , piece_id=RFLBOCME4C577HCEZUMYYPUGXYIZLUTZV55QDXFVSO2NTZWVMXGA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899444932 (559h17m18.770527977s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899449651 (559h17m18.770532686s) [noise=true , address=145.79.227.34:4907 , piece_id=3JPKUY6SG5NG562C4LZFK5POOA5MTUGTLUL5NYTUTBB7BXCMOWDA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899590659 (559h17m18.770673694s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 28 22:46:18.899601921 (559h17m18.770684966s) [noise=true , address=145.79.227.35:4481 , piece_id=TTL6Z7AJUWHLE3XIBQ4I6IUIGP7JCN3JGK5HIP42DDZ6XKNYFJUA, seg_pos={0 0}, upload=490322]: Long tail cancellation due to optimal threshold Dec 28 22:46:18.899621057 (559h17m18.770704092s) [seg_pos={0 0}, upload=490322]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:46:18.899821439 (559h17m18.770904474s) [upload=490322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:46:18.899827490 (559h17m18.770910535s) [upload=490322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:46:19.190656313 (559h17m19.061739348s) [upload=490322]: Done uploading segments... Dec 28 22:46:19.190698935 (559h17m19.06178197s) [upload=490322]: Upload finished. err: <nil> Duration: 30.00276714s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.X...................X.............] duration: 26.004364106s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:46:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 26.004381559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................X....................X.....X....X.X...X....] duration: 13.677880953s trace: https://jaeger.dev.storj.io/trace/2b5da364eaa6465e FAIL (2025-12-28 22:46:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 22:42:42.792652746 (559h13m42.663735781s) [upload=490273]: Starting upload Dec 28 22:42:42.792736115 (559h13m42.663819151s) [upload=490273]: Done starting upload Dec 28 22:42:42.792817600 (559h13m42.663900635s) [upload=490273]: Uploading segments... Dec 28 22:42:42.792973306 (559h13m42.664056341s) [upload=490273]: Got next segment. Inline: false Dec 28 22:42:42.793015326 (559h13m42.664098361s) [upload=490273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:42:42.793021449 (559h13m42.664104484s) [upload=490273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:42:43.034451327 (559h13m42.905534372s) [seg_pos={0 0}, upload=490273]: Begin upload segment... Dec 28 22:42:43.034518154 (559h13m42.905601199s) [seg_pos={0 0}, upload=490273]: Done begin upload segment. Dec 28 22:42:43.034584179 (559h13m42.905667214s) [noise=true , address=145.79.227.51:4409 , piece_id=JHUUUT7R2SV4YO5PJ3KWOYOK6VZ46MGUYMKIZOLCX2YTJS6QM24Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.034609397 (559h13m42.905692442s) [noise=true , address=145.79.227.15:4876 , piece_id=4K3P6NJHJBJLEAL2KWAUVJ5FZ5EZZBIYISTL5Y7NSKHLL77GUBDQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.035245698 (559h13m42.906328743s) [noise=true , address=145.79.227.6:4257 , piece_id=P4M53QLUDYGPO3AVOAHW44DOU4TNBSZOW75SI4YCG5LPQ6PHS6DQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.035646170 (559h13m42.906729215s) [noise=true , address=145.79.227.19:4078 , piece_id=MTIJRIYR6NJ4EMKZTHK3SQYO7JP3VNG566K2VUWFFMV2W7HP2Z5Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.036055480 (559h13m42.907138515s) [noise=true , address=45.91.116.7:4010 , piece_id=MWQ434WLSAQD2HMRNKHDKF3CXX5VEMQYBB2M6WRHCKQUPKKJBOHQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.036493614 (559h13m42.907576659s) [noise=true , address=145.79.227.13:4803 , piece_id=46HOU2RVDUBAANSTJTDVIXPEVKDTZK3EMP5W2N3QSGXRM3A6J6MQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037156265 (559h13m42.908239301s) [noise=true , address=77.90.31.18:4808 , piece_id=GZ5NU4MQJBP4RFEIRUVCOXBH4NRQFBADLINHX5BOR6IDYSUYKJXQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037176974 (559h13m42.908260009s) [noise=true , address=77.90.31.13:4627 , piece_id=LVJZDX5L4CD6CJW3TKNHICCHNVQRWXRD5TX7VA6NGC64FRHHKO7Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037564382 (559h13m42.908647417s) [noise=true , address=77.90.31.14:4662 , piece_id=Y44AXTZHBOSKVA5EJ53JS3C23UHQR7ZCSLYQ62Y42ZHWDWZ2QRQA, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037736870 (559h13m42.908819915s) [noise=true , address=45.91.116.12:4226 , piece_id=WG5HYFUZSA6H376ZQ52NNIUE3YEEHA4XC5HQTFA2PF2T7B7SSOJA, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.038328076 (559h13m42.909411111s) [noise=true , address=45.91.116.14:4190 , piece_id=YFMKA36RR5HQUYIFE3MP7FKX453KCB5QBLIIH7ZTV2MK4GCILTQQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.038815283 (559h13m42.909898318s) [noise=true , address=45.91.116.10:4083 , piece_id=62OTEKHKQEKAUR6YG546TJJUIQGCD3RKXBICAUINYBQSH25ZZOSQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.039292883 (559h13m42.910375928s) [noise=true , address=77.90.31.10:4517 , piece_id=ULOEZUXSRPMT27O3XAQN7ZQYNEC76EOSC5AWRK4JBCUCY2AOYW4Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.039780090 (559h13m42.910863125s) [noise=true , address=145.79.227.42:4123 , piece_id=WFMNCDWVMROJRYPSTVOLAFDEKDNHXDY44RHAZHDWIEKMWVZ2SXTQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.040412063 (559h13m42.911495108s) [noise=true , address=145.79.227.38:4979 , piece_id=2IOMQVVD4H5VGWKOKC5PDV5G2QBACHHTNXDRUTEBIW65GW3UKAIA, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.047313641 (559h13m42.918396686s) [noise=true , address=45.91.116.6:4974 , piece_id=GABSOBYQXXVDZM7Q67UWTNC2INPS2DU5IMB2G6UM7TKUZHBQYXNQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:44.741372689 (559h13m44.612455724s) [upload=490273]: Next returned nil segment Dec 28 22:42:44.741403137 (559h13m44.612486172s) [upload=490273]: Waiting for error group managing segments... Dec 28 22:42:45.027304600 (559h13m44.898387635s) [noise=true , address=145.79.227.6:4257 , piece_id=P4M53QLUDYGPO3AVOAHW44DOU4TNBSZOW75SI4YCG5LPQ6PHS6DQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.032728655 (559h13m44.90381169s) [noise=true , address=145.79.227.15:4876 , piece_id=4K3P6NJHJBJLEAL2KWAUVJ5FZ5EZZBIYISTL5Y7NSKHLL77GUBDQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.053784998 (559h13m44.924868043s) [noise=true , address=145.79.227.19:4078 , piece_id=MTIJRIYR6NJ4EMKZTHK3SQYO7JP3VNG566K2VUWFFMV2W7HP2Z5Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.060101611 (559h13m44.931184646s) [noise=true , address=145.79.227.51:4409 , piece_id=JHUUUT7R2SV4YO5PJ3KWOYOK6VZ46MGUYMKIZOLCX2YTJS6QM24Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.066101783 (559h13m44.937184818s) [noise=true , address=145.79.227.38:4979 , piece_id=2IOMQVVD4H5VGWKOKC5PDV5G2QBACHHTNXDRUTEBIW65GW3UKAIA, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.092430371 (559h13m44.963513406s) [noise=true , address=145.79.227.42:4123 , piece_id=WFMNCDWVMROJRYPSTVOLAFDEKDNHXDY44RHAZHDWIEKMWVZ2SXTQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.106433417 (559h13m44.977516462s) [noise=true , address=145.79.227.13:4803 , piece_id=46HOU2RVDUBAANSTJTDVIXPEVKDTZK3EMP5W2N3QSGXRM3A6J6MQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.226309259 (559h13m45.097392294s) [noise=true , address=77.90.31.14:4662 , piece_id=Y44AXTZHBOSKVA5EJ53JS3C23UHQR7ZCSLYQ62Y42ZHWDWZ2QRQA, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.259740437 (559h13m45.130823472s) [noise=true , address=45.91.116.7:4010 , piece_id=MWQ434WLSAQD2HMRNKHDKF3CXX5VEMQYBB2M6WRHCKQUPKKJBOHQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.275536778 (559h13m45.146619813s) [noise=true , address=77.90.31.18:4808 , piece_id=GZ5NU4MQJBP4RFEIRUVCOXBH4NRQFBADLINHX5BOR6IDYSUYKJXQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.291544058 (559h13m45.162627103s) [noise=true , address=77.90.31.10:4517 , piece_id=ULOEZUXSRPMT27O3XAQN7ZQYNEC76EOSC5AWRK4JBCUCY2AOYW4Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.327271488 (559h13m45.198354563s) [noise=true , address=77.90.31.13:4627 , piece_id=LVJZDX5L4CD6CJW3TKNHICCHNVQRWXRD5TX7VA6NGC64FRHHKO7Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.368718174 (559h13m45.239801209s) [noise=true , address=45.91.116.12:4226 , piece_id=WG5HYFUZSA6H376ZQ52NNIUE3YEEHA4XC5HQTFA2PF2T7B7SSOJA, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.368738173 (559h13m45.239821208s) [seg_pos={0 0}, upload=490273]: Segment reached optimal threshold of 13 pieces. Dec 28 22:42:45.368983550 (559h13m45.240066585s) [noise=true , address=45.91.116.10:4083 , piece_id=62OTEKHKQEKAUR6YG546TJJUIQGCD3RKXBICAUINYBQSH25ZZOSQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 28 22:42:45.368990062 (559h13m45.240073097s) [noise=true , address=45.91.116.10:4083 , piece_id=62OTEKHKQEKAUR6YG546TJJUIQGCD3RKXBICAUINYBQSH25ZZOSQ, seg_pos={0 0}, upload=490273]: Long tail cancellation due to optimal threshold Dec 28 22:42:45.369209349 (559h13m45.240292385s) [noise=true , address=45.91.116.6:4974 , piece_id=GABSOBYQXXVDZM7Q67UWTNC2INPS2DU5IMB2G6UM7TKUZHBQYXNQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:42:45.369214579 (559h13m45.240297614s) [noise=true , address=45.91.116.6:4974 , piece_id=GABSOBYQXXVDZM7Q67UWTNC2INPS2DU5IMB2G6UM7TKUZHBQYXNQ, seg_pos={0 0}, upload=490273]: Long tail cancellation due to optimal threshold Dec 28 22:42:45.369314249 (559h13m45.240397284s) [noise=true , address=45.91.116.14:4190 , piece_id=YFMKA36RR5HQUYIFE3MP7FKX453KCB5QBLIIH7ZTV2MK4GCILTQQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 22:42:45.369321673 (559h13m45.240404708s) [noise=true , address=45.91.116.14:4190 , piece_id=YFMKA36RR5HQUYIFE3MP7FKX453KCB5QBLIIH7ZTV2MK4GCILTQQ, seg_pos={0 0}, upload=490273]: Long tail cancellation due to optimal threshold Dec 28 22:42:45.369337073 (559h13m45.240420108s) [seg_pos={0 0}, upload=490273]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:42:45.369401304 (559h13m45.24048434s) [upload=490273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:42:45.369406916 (559h13m45.240489951s) [upload=490273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:42:45.661510754 (559h13m45.532593789s) [upload=490273]: Done uploading segments... Dec 28 22:42:45.661549378 (559h13m45.532632423s) [upload=490273]: Upload finished. err: <nil> Duration: 13.677913694s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.X...................X............] duration: 15.32564268s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:45:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 15.325670693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.X...................X...........] duration: 16.103239882s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:45:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 16.103269579s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................X....................X.....X....X.X...X...] duration: 8.950943205s trace: https://jaeger.dev.storj.io/trace/2b5da364eaa6465e FAIL (2025-12-28 22:44:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 22:42:42.792652746 (559h13m42.663735781s) [upload=490273]: Starting upload Dec 28 22:42:42.792736115 (559h13m42.663819151s) [upload=490273]: Done starting upload Dec 28 22:42:42.792817600 (559h13m42.663900635s) [upload=490273]: Uploading segments... Dec 28 22:42:42.792973306 (559h13m42.664056341s) [upload=490273]: Got next segment. Inline: false Dec 28 22:42:42.793015326 (559h13m42.664098361s) [upload=490273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:42:42.793021449 (559h13m42.664104484s) [upload=490273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:42:43.034451327 (559h13m42.905534372s) [seg_pos={0 0}, upload=490273]: Begin upload segment... Dec 28 22:42:43.034518154 (559h13m42.905601199s) [seg_pos={0 0}, upload=490273]: Done begin upload segment. Dec 28 22:42:43.034584179 (559h13m42.905667214s) [noise=true , address=145.79.227.51:4409 , piece_id=JHUUUT7R2SV4YO5PJ3KWOYOK6VZ46MGUYMKIZOLCX2YTJS6QM24Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.034609397 (559h13m42.905692442s) [noise=true , address=145.79.227.15:4876 , piece_id=4K3P6NJHJBJLEAL2KWAUVJ5FZ5EZZBIYISTL5Y7NSKHLL77GUBDQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.035245698 (559h13m42.906328743s) [noise=true , address=145.79.227.6:4257 , piece_id=P4M53QLUDYGPO3AVOAHW44DOU4TNBSZOW75SI4YCG5LPQ6PHS6DQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.035646170 (559h13m42.906729215s) [noise=true , address=145.79.227.19:4078 , piece_id=MTIJRIYR6NJ4EMKZTHK3SQYO7JP3VNG566K2VUWFFMV2W7HP2Z5Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.036055480 (559h13m42.907138515s) [noise=true , address=45.91.116.7:4010 , piece_id=MWQ434WLSAQD2HMRNKHDKF3CXX5VEMQYBB2M6WRHCKQUPKKJBOHQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.036493614 (559h13m42.907576659s) [noise=true , address=145.79.227.13:4803 , piece_id=46HOU2RVDUBAANSTJTDVIXPEVKDTZK3EMP5W2N3QSGXRM3A6J6MQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037156265 (559h13m42.908239301s) [noise=true , address=77.90.31.18:4808 , piece_id=GZ5NU4MQJBP4RFEIRUVCOXBH4NRQFBADLINHX5BOR6IDYSUYKJXQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037176974 (559h13m42.908260009s) [noise=true , address=77.90.31.13:4627 , piece_id=LVJZDX5L4CD6CJW3TKNHICCHNVQRWXRD5TX7VA6NGC64FRHHKO7Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037564382 (559h13m42.908647417s) [noise=true , address=77.90.31.14:4662 , piece_id=Y44AXTZHBOSKVA5EJ53JS3C23UHQR7ZCSLYQ62Y42ZHWDWZ2QRQA, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037736870 (559h13m42.908819915s) [noise=true , address=45.91.116.12:4226 , piece_id=WG5HYFUZSA6H376ZQ52NNIUE3YEEHA4XC5HQTFA2PF2T7B7SSOJA, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.038328076 (559h13m42.909411111s) [noise=true , address=45.91.116.14:4190 , piece_id=YFMKA36RR5HQUYIFE3MP7FKX453KCB5QBLIIH7ZTV2MK4GCILTQQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.038815283 (559h13m42.909898318s) [noise=true , address=45.91.116.10:4083 , piece_id=62OTEKHKQEKAUR6YG546TJJUIQGCD3RKXBICAUINYBQSH25ZZOSQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.039292883 (559h13m42.910375928s) [noise=true , address=77.90.31.10:4517 , piece_id=ULOEZUXSRPMT27O3XAQN7ZQYNEC76EOSC5AWRK4JBCUCY2AOYW4Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.039780090 (559h13m42.910863125s) [noise=true , address=145.79.227.42:4123 , piece_id=WFMNCDWVMROJRYPSTVOLAFDEKDNHXDY44RHAZHDWIEKMWVZ2SXTQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.040412063 (559h13m42.911495108s) [noise=true , address=145.79.227.38:4979 , piece_id=2IOMQVVD4H5VGWKOKC5PDV5G2QBACHHTNXDRUTEBIW65GW3UKAIA, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.047313641 (559h13m42.918396686s) [noise=true , address=45.91.116.6:4974 , piece_id=GABSOBYQXXVDZM7Q67UWTNC2INPS2DU5IMB2G6UM7TKUZHBQYXNQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:44.741372689 (559h13m44.612455724s) [upload=490273]: Next returned nil segment Dec 28 22:42:44.741403137 (559h13m44.612486172s) [upload=490273]: Waiting for error group managing segments... Dec 28 22:42:45.027304600 (559h13m44.898387635s) [noise=true , address=145.79.227.6:4257 , piece_id=P4M53QLUDYGPO3AVOAHW44DOU4TNBSZOW75SI4YCG5LPQ6PHS6DQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.032728655 (559h13m44.90381169s) [noise=true , address=145.79.227.15:4876 , piece_id=4K3P6NJHJBJLEAL2KWAUVJ5FZ5EZZBIYISTL5Y7NSKHLL77GUBDQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.053784998 (559h13m44.924868043s) [noise=true , address=145.79.227.19:4078 , piece_id=MTIJRIYR6NJ4EMKZTHK3SQYO7JP3VNG566K2VUWFFMV2W7HP2Z5Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.060101611 (559h13m44.931184646s) [noise=true , address=145.79.227.51:4409 , piece_id=JHUUUT7R2SV4YO5PJ3KWOYOK6VZ46MGUYMKIZOLCX2YTJS6QM24Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.066101783 (559h13m44.937184818s) [noise=true , address=145.79.227.38:4979 , piece_id=2IOMQVVD4H5VGWKOKC5PDV5G2QBACHHTNXDRUTEBIW65GW3UKAIA, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.092430371 (559h13m44.963513406s) [noise=true , address=145.79.227.42:4123 , piece_id=WFMNCDWVMROJRYPSTVOLAFDEKDNHXDY44RHAZHDWIEKMWVZ2SXTQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.106433417 (559h13m44.977516462s) [noise=true , address=145.79.227.13:4803 , piece_id=46HOU2RVDUBAANSTJTDVIXPEVKDTZK3EMP5W2N3QSGXRM3A6J6MQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.226309259 (559h13m45.097392294s) [noise=true , address=77.90.31.14:4662 , piece_id=Y44AXTZHBOSKVA5EJ53JS3C23UHQR7ZCSLYQ62Y42ZHWDWZ2QRQA, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.259740437 (559h13m45.130823472s) [noise=true , address=45.91.116.7:4010 , piece_id=MWQ434WLSAQD2HMRNKHDKF3CXX5VEMQYBB2M6WRHCKQUPKKJBOHQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.275536778 (559h13m45.146619813s) [noise=true , address=77.90.31.18:4808 , piece_id=GZ5NU4MQJBP4RFEIRUVCOXBH4NRQFBADLINHX5BOR6IDYSUYKJXQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.291544058 (559h13m45.162627103s) [noise=true , address=77.90.31.10:4517 , piece_id=ULOEZUXSRPMT27O3XAQN7ZQYNEC76EOSC5AWRK4JBCUCY2AOYW4Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.327271488 (559h13m45.198354563s) [noise=true , address=77.90.31.13:4627 , piece_id=LVJZDX5L4CD6CJW3TKNHICCHNVQRWXRD5TX7VA6NGC64FRHHKO7Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.368718174 (559h13m45.239801209s) [noise=true , address=45.91.116.12:4226 , piece_id=WG5HYFUZSA6H376ZQ52NNIUE3YEEHA4XC5HQTFA2PF2T7B7SSOJA, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.368738173 (559h13m45.239821208s) [seg_pos={0 0}, upload=490273]: Segment reached optimal threshold of 13 pieces. Dec 28 22:42:45.368983550 (559h13m45.240066585s) [noise=true , address=45.91.116.10:4083 , piece_id=62OTEKHKQEKAUR6YG546TJJUIQGCD3RKXBICAUINYBQSH25ZZOSQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 28 22:42:45.368990062 (559h13m45.240073097s) [noise=true , address=45.91.116.10:4083 , piece_id=62OTEKHKQEKAUR6YG546TJJUIQGCD3RKXBICAUINYBQSH25ZZOSQ, seg_pos={0 0}, upload=490273]: Long tail cancellation due to optimal threshold Dec 28 22:42:45.369209349 (559h13m45.240292385s) [noise=true , address=45.91.116.6:4974 , piece_id=GABSOBYQXXVDZM7Q67UWTNC2INPS2DU5IMB2G6UM7TKUZHBQYXNQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:42:45.369214579 (559h13m45.240297614s) [noise=true , address=45.91.116.6:4974 , piece_id=GABSOBYQXXVDZM7Q67UWTNC2INPS2DU5IMB2G6UM7TKUZHBQYXNQ, seg_pos={0 0}, upload=490273]: Long tail cancellation due to optimal threshold Dec 28 22:42:45.369314249 (559h13m45.240397284s) [noise=true , address=45.91.116.14:4190 , piece_id=YFMKA36RR5HQUYIFE3MP7FKX453KCB5QBLIIH7ZTV2MK4GCILTQQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 22:42:45.369321673 (559h13m45.240404708s) [noise=true , address=45.91.116.14:4190 , piece_id=YFMKA36RR5HQUYIFE3MP7FKX453KCB5QBLIIH7ZTV2MK4GCILTQQ, seg_pos={0 0}, upload=490273]: Long tail cancellation due to optimal threshold Dec 28 22:42:45.369337073 (559h13m45.240420108s) [seg_pos={0 0}, upload=490273]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:42:45.369401304 (559h13m45.24048434s) [upload=490273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:42:45.369406916 (559h13m45.240489951s) [upload=490273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:42:45.661510754 (559h13m45.532593789s) [upload=490273]: Done uploading segments... Dec 28 22:42:45.661549378 (559h13m45.532632423s) [upload=490273]: Upload finished. err: <nil> Duration: 8.950968242s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.X...................X..........] duration: 22.474468553s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:44:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 22.474505645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................X....................X.....X....X.X...X..] duration: 7.040191921s trace: https://jaeger.dev.storj.io/trace/2b5da364eaa6465e FAIL (2025-12-28 22:43:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 22:42:42.792652746 (559h13m42.663735781s) [upload=490273]: Starting upload Dec 28 22:42:42.792736115 (559h13m42.663819151s) [upload=490273]: Done starting upload Dec 28 22:42:42.792817600 (559h13m42.663900635s) [upload=490273]: Uploading segments... Dec 28 22:42:42.792973306 (559h13m42.664056341s) [upload=490273]: Got next segment. Inline: false Dec 28 22:42:42.793015326 (559h13m42.664098361s) [upload=490273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:42:42.793021449 (559h13m42.664104484s) [upload=490273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:42:43.034451327 (559h13m42.905534372s) [seg_pos={0 0}, upload=490273]: Begin upload segment... Dec 28 22:42:43.034518154 (559h13m42.905601199s) [seg_pos={0 0}, upload=490273]: Done begin upload segment. Dec 28 22:42:43.034584179 (559h13m42.905667214s) [noise=true , address=145.79.227.51:4409 , piece_id=JHUUUT7R2SV4YO5PJ3KWOYOK6VZ46MGUYMKIZOLCX2YTJS6QM24Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.034609397 (559h13m42.905692442s) [noise=true , address=145.79.227.15:4876 , piece_id=4K3P6NJHJBJLEAL2KWAUVJ5FZ5EZZBIYISTL5Y7NSKHLL77GUBDQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.035245698 (559h13m42.906328743s) [noise=true , address=145.79.227.6:4257 , piece_id=P4M53QLUDYGPO3AVOAHW44DOU4TNBSZOW75SI4YCG5LPQ6PHS6DQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.035646170 (559h13m42.906729215s) [noise=true , address=145.79.227.19:4078 , piece_id=MTIJRIYR6NJ4EMKZTHK3SQYO7JP3VNG566K2VUWFFMV2W7HP2Z5Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.036055480 (559h13m42.907138515s) [noise=true , address=45.91.116.7:4010 , piece_id=MWQ434WLSAQD2HMRNKHDKF3CXX5VEMQYBB2M6WRHCKQUPKKJBOHQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.036493614 (559h13m42.907576659s) [noise=true , address=145.79.227.13:4803 , piece_id=46HOU2RVDUBAANSTJTDVIXPEVKDTZK3EMP5W2N3QSGXRM3A6J6MQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037156265 (559h13m42.908239301s) [noise=true , address=77.90.31.18:4808 , piece_id=GZ5NU4MQJBP4RFEIRUVCOXBH4NRQFBADLINHX5BOR6IDYSUYKJXQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037176974 (559h13m42.908260009s) [noise=true , address=77.90.31.13:4627 , piece_id=LVJZDX5L4CD6CJW3TKNHICCHNVQRWXRD5TX7VA6NGC64FRHHKO7Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037564382 (559h13m42.908647417s) [noise=true , address=77.90.31.14:4662 , piece_id=Y44AXTZHBOSKVA5EJ53JS3C23UHQR7ZCSLYQ62Y42ZHWDWZ2QRQA, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037736870 (559h13m42.908819915s) [noise=true , address=45.91.116.12:4226 , piece_id=WG5HYFUZSA6H376ZQ52NNIUE3YEEHA4XC5HQTFA2PF2T7B7SSOJA, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.038328076 (559h13m42.909411111s) [noise=true , address=45.91.116.14:4190 , piece_id=YFMKA36RR5HQUYIFE3MP7FKX453KCB5QBLIIH7ZTV2MK4GCILTQQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.038815283 (559h13m42.909898318s) [noise=true , address=45.91.116.10:4083 , piece_id=62OTEKHKQEKAUR6YG546TJJUIQGCD3RKXBICAUINYBQSH25ZZOSQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.039292883 (559h13m42.910375928s) [noise=true , address=77.90.31.10:4517 , piece_id=ULOEZUXSRPMT27O3XAQN7ZQYNEC76EOSC5AWRK4JBCUCY2AOYW4Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.039780090 (559h13m42.910863125s) [noise=true , address=145.79.227.42:4123 , piece_id=WFMNCDWVMROJRYPSTVOLAFDEKDNHXDY44RHAZHDWIEKMWVZ2SXTQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.040412063 (559h13m42.911495108s) [noise=true , address=145.79.227.38:4979 , piece_id=2IOMQVVD4H5VGWKOKC5PDV5G2QBACHHTNXDRUTEBIW65GW3UKAIA, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.047313641 (559h13m42.918396686s) [noise=true , address=45.91.116.6:4974 , piece_id=GABSOBYQXXVDZM7Q67UWTNC2INPS2DU5IMB2G6UM7TKUZHBQYXNQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:44.741372689 (559h13m44.612455724s) [upload=490273]: Next returned nil segment Dec 28 22:42:44.741403137 (559h13m44.612486172s) [upload=490273]: Waiting for error group managing segments... Dec 28 22:42:45.027304600 (559h13m44.898387635s) [noise=true , address=145.79.227.6:4257 , piece_id=P4M53QLUDYGPO3AVOAHW44DOU4TNBSZOW75SI4YCG5LPQ6PHS6DQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.032728655 (559h13m44.90381169s) [noise=true , address=145.79.227.15:4876 , piece_id=4K3P6NJHJBJLEAL2KWAUVJ5FZ5EZZBIYISTL5Y7NSKHLL77GUBDQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.053784998 (559h13m44.924868043s) [noise=true , address=145.79.227.19:4078 , piece_id=MTIJRIYR6NJ4EMKZTHK3SQYO7JP3VNG566K2VUWFFMV2W7HP2Z5Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.060101611 (559h13m44.931184646s) [noise=true , address=145.79.227.51:4409 , piece_id=JHUUUT7R2SV4YO5PJ3KWOYOK6VZ46MGUYMKIZOLCX2YTJS6QM24Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.066101783 (559h13m44.937184818s) [noise=true , address=145.79.227.38:4979 , piece_id=2IOMQVVD4H5VGWKOKC5PDV5G2QBACHHTNXDRUTEBIW65GW3UKAIA, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.092430371 (559h13m44.963513406s) [noise=true , address=145.79.227.42:4123 , piece_id=WFMNCDWVMROJRYPSTVOLAFDEKDNHXDY44RHAZHDWIEKMWVZ2SXTQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.106433417 (559h13m44.977516462s) [noise=true , address=145.79.227.13:4803 , piece_id=46HOU2RVDUBAANSTJTDVIXPEVKDTZK3EMP5W2N3QSGXRM3A6J6MQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.226309259 (559h13m45.097392294s) [noise=true , address=77.90.31.14:4662 , piece_id=Y44AXTZHBOSKVA5EJ53JS3C23UHQR7ZCSLYQ62Y42ZHWDWZ2QRQA, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.259740437 (559h13m45.130823472s) [noise=true , address=45.91.116.7:4010 , piece_id=MWQ434WLSAQD2HMRNKHDKF3CXX5VEMQYBB2M6WRHCKQUPKKJBOHQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.275536778 (559h13m45.146619813s) [noise=true , address=77.90.31.18:4808 , piece_id=GZ5NU4MQJBP4RFEIRUVCOXBH4NRQFBADLINHX5BOR6IDYSUYKJXQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.291544058 (559h13m45.162627103s) [noise=true , address=77.90.31.10:4517 , piece_id=ULOEZUXSRPMT27O3XAQN7ZQYNEC76EOSC5AWRK4JBCUCY2AOYW4Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.327271488 (559h13m45.198354563s) [noise=true , address=77.90.31.13:4627 , piece_id=LVJZDX5L4CD6CJW3TKNHICCHNVQRWXRD5TX7VA6NGC64FRHHKO7Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.368718174 (559h13m45.239801209s) [noise=true , address=45.91.116.12:4226 , piece_id=WG5HYFUZSA6H376ZQ52NNIUE3YEEHA4XC5HQTFA2PF2T7B7SSOJA, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.368738173 (559h13m45.239821208s) [seg_pos={0 0}, upload=490273]: Segment reached optimal threshold of 13 pieces. Dec 28 22:42:45.368983550 (559h13m45.240066585s) [noise=true , address=45.91.116.10:4083 , piece_id=62OTEKHKQEKAUR6YG546TJJUIQGCD3RKXBICAUINYBQSH25ZZOSQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 28 22:42:45.368990062 (559h13m45.240073097s) [noise=true , address=45.91.116.10:4083 , piece_id=62OTEKHKQEKAUR6YG546TJJUIQGCD3RKXBICAUINYBQSH25ZZOSQ, seg_pos={0 0}, upload=490273]: Long tail cancellation due to optimal threshold Dec 28 22:42:45.369209349 (559h13m45.240292385s) [noise=true , address=45.91.116.6:4974 , piece_id=GABSOBYQXXVDZM7Q67UWTNC2INPS2DU5IMB2G6UM7TKUZHBQYXNQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:42:45.369214579 (559h13m45.240297614s) [noise=true , address=45.91.116.6:4974 , piece_id=GABSOBYQXXVDZM7Q67UWTNC2INPS2DU5IMB2G6UM7TKUZHBQYXNQ, seg_pos={0 0}, upload=490273]: Long tail cancellation due to optimal threshold Dec 28 22:42:45.369314249 (559h13m45.240397284s) [noise=true , address=45.91.116.14:4190 , piece_id=YFMKA36RR5HQUYIFE3MP7FKX453KCB5QBLIIH7ZTV2MK4GCILTQQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 22:42:45.369321673 (559h13m45.240404708s) [noise=true , address=45.91.116.14:4190 , piece_id=YFMKA36RR5HQUYIFE3MP7FKX453KCB5QBLIIH7ZTV2MK4GCILTQQ, seg_pos={0 0}, upload=490273]: Long tail cancellation due to optimal threshold Dec 28 22:42:45.369337073 (559h13m45.240420108s) [seg_pos={0 0}, upload=490273]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:42:45.369401304 (559h13m45.24048434s) [upload=490273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:42:45.369406916 (559h13m45.240489951s) [upload=490273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:42:45.661510754 (559h13m45.532593789s) [upload=490273]: Done uploading segments... Dec 28 22:42:45.661549378 (559h13m45.532632423s) [upload=490273]: Upload finished. err: <nil> Duration: 7.040220526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X.X...................X.........] duration: 23.710354904s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:43:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 23.710381835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................X....................X.....X....X.X...X.] duration: 6.003212598s trace: https://jaeger.dev.storj.io/trace/2b5da364eaa6465e FAIL (2025-12-28 22:43:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 22:42:42.792652746 (559h13m42.663735781s) [upload=490273]: Starting upload Dec 28 22:42:42.792736115 (559h13m42.663819151s) [upload=490273]: Done starting upload Dec 28 22:42:42.792817600 (559h13m42.663900635s) [upload=490273]: Uploading segments... Dec 28 22:42:42.792973306 (559h13m42.664056341s) [upload=490273]: Got next segment. Inline: false Dec 28 22:42:42.793015326 (559h13m42.664098361s) [upload=490273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:42:42.793021449 (559h13m42.664104484s) [upload=490273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:42:43.034451327 (559h13m42.905534372s) [seg_pos={0 0}, upload=490273]: Begin upload segment... Dec 28 22:42:43.034518154 (559h13m42.905601199s) [seg_pos={0 0}, upload=490273]: Done begin upload segment. Dec 28 22:42:43.034584179 (559h13m42.905667214s) [noise=true , address=145.79.227.51:4409 , piece_id=JHUUUT7R2SV4YO5PJ3KWOYOK6VZ46MGUYMKIZOLCX2YTJS6QM24Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.034609397 (559h13m42.905692442s) [noise=true , address=145.79.227.15:4876 , piece_id=4K3P6NJHJBJLEAL2KWAUVJ5FZ5EZZBIYISTL5Y7NSKHLL77GUBDQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.035245698 (559h13m42.906328743s) [noise=true , address=145.79.227.6:4257 , piece_id=P4M53QLUDYGPO3AVOAHW44DOU4TNBSZOW75SI4YCG5LPQ6PHS6DQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.035646170 (559h13m42.906729215s) [noise=true , address=145.79.227.19:4078 , piece_id=MTIJRIYR6NJ4EMKZTHK3SQYO7JP3VNG566K2VUWFFMV2W7HP2Z5Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.036055480 (559h13m42.907138515s) [noise=true , address=45.91.116.7:4010 , piece_id=MWQ434WLSAQD2HMRNKHDKF3CXX5VEMQYBB2M6WRHCKQUPKKJBOHQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.036493614 (559h13m42.907576659s) [noise=true , address=145.79.227.13:4803 , piece_id=46HOU2RVDUBAANSTJTDVIXPEVKDTZK3EMP5W2N3QSGXRM3A6J6MQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037156265 (559h13m42.908239301s) [noise=true , address=77.90.31.18:4808 , piece_id=GZ5NU4MQJBP4RFEIRUVCOXBH4NRQFBADLINHX5BOR6IDYSUYKJXQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037176974 (559h13m42.908260009s) [noise=true , address=77.90.31.13:4627 , piece_id=LVJZDX5L4CD6CJW3TKNHICCHNVQRWXRD5TX7VA6NGC64FRHHKO7Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037564382 (559h13m42.908647417s) [noise=true , address=77.90.31.14:4662 , piece_id=Y44AXTZHBOSKVA5EJ53JS3C23UHQR7ZCSLYQ62Y42ZHWDWZ2QRQA, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037736870 (559h13m42.908819915s) [noise=true , address=45.91.116.12:4226 , piece_id=WG5HYFUZSA6H376ZQ52NNIUE3YEEHA4XC5HQTFA2PF2T7B7SSOJA, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.038328076 (559h13m42.909411111s) [noise=true , address=45.91.116.14:4190 , piece_id=YFMKA36RR5HQUYIFE3MP7FKX453KCB5QBLIIH7ZTV2MK4GCILTQQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.038815283 (559h13m42.909898318s) [noise=true , address=45.91.116.10:4083 , piece_id=62OTEKHKQEKAUR6YG546TJJUIQGCD3RKXBICAUINYBQSH25ZZOSQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.039292883 (559h13m42.910375928s) [noise=true , address=77.90.31.10:4517 , piece_id=ULOEZUXSRPMT27O3XAQN7ZQYNEC76EOSC5AWRK4JBCUCY2AOYW4Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.039780090 (559h13m42.910863125s) [noise=true , address=145.79.227.42:4123 , piece_id=WFMNCDWVMROJRYPSTVOLAFDEKDNHXDY44RHAZHDWIEKMWVZ2SXTQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.040412063 (559h13m42.911495108s) [noise=true , address=145.79.227.38:4979 , piece_id=2IOMQVVD4H5VGWKOKC5PDV5G2QBACHHTNXDRUTEBIW65GW3UKAIA, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.047313641 (559h13m42.918396686s) [noise=true , address=45.91.116.6:4974 , piece_id=GABSOBYQXXVDZM7Q67UWTNC2INPS2DU5IMB2G6UM7TKUZHBQYXNQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:44.741372689 (559h13m44.612455724s) [upload=490273]: Next returned nil segment Dec 28 22:42:44.741403137 (559h13m44.612486172s) [upload=490273]: Waiting for error group managing segments... Dec 28 22:42:45.027304600 (559h13m44.898387635s) [noise=true , address=145.79.227.6:4257 , piece_id=P4M53QLUDYGPO3AVOAHW44DOU4TNBSZOW75SI4YCG5LPQ6PHS6DQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.032728655 (559h13m44.90381169s) [noise=true , address=145.79.227.15:4876 , piece_id=4K3P6NJHJBJLEAL2KWAUVJ5FZ5EZZBIYISTL5Y7NSKHLL77GUBDQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.053784998 (559h13m44.924868043s) [noise=true , address=145.79.227.19:4078 , piece_id=MTIJRIYR6NJ4EMKZTHK3SQYO7JP3VNG566K2VUWFFMV2W7HP2Z5Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.060101611 (559h13m44.931184646s) [noise=true , address=145.79.227.51:4409 , piece_id=JHUUUT7R2SV4YO5PJ3KWOYOK6VZ46MGUYMKIZOLCX2YTJS6QM24Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.066101783 (559h13m44.937184818s) [noise=true , address=145.79.227.38:4979 , piece_id=2IOMQVVD4H5VGWKOKC5PDV5G2QBACHHTNXDRUTEBIW65GW3UKAIA, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.092430371 (559h13m44.963513406s) [noise=true , address=145.79.227.42:4123 , piece_id=WFMNCDWVMROJRYPSTVOLAFDEKDNHXDY44RHAZHDWIEKMWVZ2SXTQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.106433417 (559h13m44.977516462s) [noise=true , address=145.79.227.13:4803 , piece_id=46HOU2RVDUBAANSTJTDVIXPEVKDTZK3EMP5W2N3QSGXRM3A6J6MQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.226309259 (559h13m45.097392294s) [noise=true , address=77.90.31.14:4662 , piece_id=Y44AXTZHBOSKVA5EJ53JS3C23UHQR7ZCSLYQ62Y42ZHWDWZ2QRQA, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.259740437 (559h13m45.130823472s) [noise=true , address=45.91.116.7:4010 , piece_id=MWQ434WLSAQD2HMRNKHDKF3CXX5VEMQYBB2M6WRHCKQUPKKJBOHQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.275536778 (559h13m45.146619813s) [noise=true , address=77.90.31.18:4808 , piece_id=GZ5NU4MQJBP4RFEIRUVCOXBH4NRQFBADLINHX5BOR6IDYSUYKJXQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.291544058 (559h13m45.162627103s) [noise=true , address=77.90.31.10:4517 , piece_id=ULOEZUXSRPMT27O3XAQN7ZQYNEC76EOSC5AWRK4JBCUCY2AOYW4Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.327271488 (559h13m45.198354563s) [noise=true , address=77.90.31.13:4627 , piece_id=LVJZDX5L4CD6CJW3TKNHICCHNVQRWXRD5TX7VA6NGC64FRHHKO7Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.368718174 (559h13m45.239801209s) [noise=true , address=45.91.116.12:4226 , piece_id=WG5HYFUZSA6H376ZQ52NNIUE3YEEHA4XC5HQTFA2PF2T7B7SSOJA, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.368738173 (559h13m45.239821208s) [seg_pos={0 0}, upload=490273]: Segment reached optimal threshold of 13 pieces. Dec 28 22:42:45.368983550 (559h13m45.240066585s) [noise=true , address=45.91.116.10:4083 , piece_id=62OTEKHKQEKAUR6YG546TJJUIQGCD3RKXBICAUINYBQSH25ZZOSQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 28 22:42:45.368990062 (559h13m45.240073097s) [noise=true , address=45.91.116.10:4083 , piece_id=62OTEKHKQEKAUR6YG546TJJUIQGCD3RKXBICAUINYBQSH25ZZOSQ, seg_pos={0 0}, upload=490273]: Long tail cancellation due to optimal threshold Dec 28 22:42:45.369209349 (559h13m45.240292385s) [noise=true , address=45.91.116.6:4974 , piece_id=GABSOBYQXXVDZM7Q67UWTNC2INPS2DU5IMB2G6UM7TKUZHBQYXNQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:42:45.369214579 (559h13m45.240297614s) [noise=true , address=45.91.116.6:4974 , piece_id=GABSOBYQXXVDZM7Q67UWTNC2INPS2DU5IMB2G6UM7TKUZHBQYXNQ, seg_pos={0 0}, upload=490273]: Long tail cancellation due to optimal threshold Dec 28 22:42:45.369314249 (559h13m45.240397284s) [noise=true , address=45.91.116.14:4190 , piece_id=YFMKA36RR5HQUYIFE3MP7FKX453KCB5QBLIIH7ZTV2MK4GCILTQQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 22:42:45.369321673 (559h13m45.240404708s) [noise=true , address=45.91.116.14:4190 , piece_id=YFMKA36RR5HQUYIFE3MP7FKX453KCB5QBLIIH7ZTV2MK4GCILTQQ, seg_pos={0 0}, upload=490273]: Long tail cancellation due to optimal threshold Dec 28 22:42:45.369337073 (559h13m45.240420108s) [seg_pos={0 0}, upload=490273]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:42:45.369401304 (559h13m45.24048434s) [upload=490273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:42:45.369406916 (559h13m45.240489951s) [upload=490273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:42:45.661510754 (559h13m45.532593789s) [upload=490273]: Done uploading segments... Dec 28 22:42:45.661549378 (559h13m45.532632423s) [upload=490273]: Upload finished. err: <nil> Duration: 6.003226594s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................X....................X.....X....X.X...X] duration: 30.003720146s trace: https://jaeger.dev.storj.io/trace/2b5da364eaa6465e FAIL (2025-12-28 22:43:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 22:42:42.792652746 (559h13m42.663735781s) [upload=490273]: Starting upload Dec 28 22:42:42.792736115 (559h13m42.663819151s) [upload=490273]: Done starting upload Dec 28 22:42:42.792817600 (559h13m42.663900635s) [upload=490273]: Uploading segments... Dec 28 22:42:42.792973306 (559h13m42.664056341s) [upload=490273]: Got next segment. Inline: false Dec 28 22:42:42.793015326 (559h13m42.664098361s) [upload=490273]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:42:42.793021449 (559h13m42.664104484s) [upload=490273]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:42:43.034451327 (559h13m42.905534372s) [seg_pos={0 0}, upload=490273]: Begin upload segment... Dec 28 22:42:43.034518154 (559h13m42.905601199s) [seg_pos={0 0}, upload=490273]: Done begin upload segment. Dec 28 22:42:43.034584179 (559h13m42.905667214s) [noise=true , address=145.79.227.51:4409 , piece_id=JHUUUT7R2SV4YO5PJ3KWOYOK6VZ46MGUYMKIZOLCX2YTJS6QM24Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.034609397 (559h13m42.905692442s) [noise=true , address=145.79.227.15:4876 , piece_id=4K3P6NJHJBJLEAL2KWAUVJ5FZ5EZZBIYISTL5Y7NSKHLL77GUBDQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.035245698 (559h13m42.906328743s) [noise=true , address=145.79.227.6:4257 , piece_id=P4M53QLUDYGPO3AVOAHW44DOU4TNBSZOW75SI4YCG5LPQ6PHS6DQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.035646170 (559h13m42.906729215s) [noise=true , address=145.79.227.19:4078 , piece_id=MTIJRIYR6NJ4EMKZTHK3SQYO7JP3VNG566K2VUWFFMV2W7HP2Z5Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.036055480 (559h13m42.907138515s) [noise=true , address=45.91.116.7:4010 , piece_id=MWQ434WLSAQD2HMRNKHDKF3CXX5VEMQYBB2M6WRHCKQUPKKJBOHQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.036493614 (559h13m42.907576659s) [noise=true , address=145.79.227.13:4803 , piece_id=46HOU2RVDUBAANSTJTDVIXPEVKDTZK3EMP5W2N3QSGXRM3A6J6MQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037156265 (559h13m42.908239301s) [noise=true , address=77.90.31.18:4808 , piece_id=GZ5NU4MQJBP4RFEIRUVCOXBH4NRQFBADLINHX5BOR6IDYSUYKJXQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037176974 (559h13m42.908260009s) [noise=true , address=77.90.31.13:4627 , piece_id=LVJZDX5L4CD6CJW3TKNHICCHNVQRWXRD5TX7VA6NGC64FRHHKO7Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037564382 (559h13m42.908647417s) [noise=true , address=77.90.31.14:4662 , piece_id=Y44AXTZHBOSKVA5EJ53JS3C23UHQR7ZCSLYQ62Y42ZHWDWZ2QRQA, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.037736870 (559h13m42.908819915s) [noise=true , address=45.91.116.12:4226 , piece_id=WG5HYFUZSA6H376ZQ52NNIUE3YEEHA4XC5HQTFA2PF2T7B7SSOJA, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.038328076 (559h13m42.909411111s) [noise=true , address=45.91.116.14:4190 , piece_id=YFMKA36RR5HQUYIFE3MP7FKX453KCB5QBLIIH7ZTV2MK4GCILTQQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.038815283 (559h13m42.909898318s) [noise=true , address=45.91.116.10:4083 , piece_id=62OTEKHKQEKAUR6YG546TJJUIQGCD3RKXBICAUINYBQSH25ZZOSQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.039292883 (559h13m42.910375928s) [noise=true , address=77.90.31.10:4517 , piece_id=ULOEZUXSRPMT27O3XAQN7ZQYNEC76EOSC5AWRK4JBCUCY2AOYW4Q, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.039780090 (559h13m42.910863125s) [noise=true , address=145.79.227.42:4123 , piece_id=WFMNCDWVMROJRYPSTVOLAFDEKDNHXDY44RHAZHDWIEKMWVZ2SXTQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.040412063 (559h13m42.911495108s) [noise=true , address=145.79.227.38:4979 , piece_id=2IOMQVVD4H5VGWKOKC5PDV5G2QBACHHTNXDRUTEBIW65GW3UKAIA, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:43.047313641 (559h13m42.918396686s) [noise=true , address=45.91.116.6:4974 , piece_id=GABSOBYQXXVDZM7Q67UWTNC2INPS2DU5IMB2G6UM7TKUZHBQYXNQ, seg_pos={0 0}, upload=490273]: Uploading piece... Dec 28 22:42:44.741372689 (559h13m44.612455724s) [upload=490273]: Next returned nil segment Dec 28 22:42:44.741403137 (559h13m44.612486172s) [upload=490273]: Waiting for error group managing segments... Dec 28 22:42:45.027304600 (559h13m44.898387635s) [noise=true , address=145.79.227.6:4257 , piece_id=P4M53QLUDYGPO3AVOAHW44DOU4TNBSZOW75SI4YCG5LPQ6PHS6DQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.032728655 (559h13m44.90381169s) [noise=true , address=145.79.227.15:4876 , piece_id=4K3P6NJHJBJLEAL2KWAUVJ5FZ5EZZBIYISTL5Y7NSKHLL77GUBDQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.053784998 (559h13m44.924868043s) [noise=true , address=145.79.227.19:4078 , piece_id=MTIJRIYR6NJ4EMKZTHK3SQYO7JP3VNG566K2VUWFFMV2W7HP2Z5Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.060101611 (559h13m44.931184646s) [noise=true , address=145.79.227.51:4409 , piece_id=JHUUUT7R2SV4YO5PJ3KWOYOK6VZ46MGUYMKIZOLCX2YTJS6QM24Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.066101783 (559h13m44.937184818s) [noise=true , address=145.79.227.38:4979 , piece_id=2IOMQVVD4H5VGWKOKC5PDV5G2QBACHHTNXDRUTEBIW65GW3UKAIA, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.092430371 (559h13m44.963513406s) [noise=true , address=145.79.227.42:4123 , piece_id=WFMNCDWVMROJRYPSTVOLAFDEKDNHXDY44RHAZHDWIEKMWVZ2SXTQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.106433417 (559h13m44.977516462s) [noise=true , address=145.79.227.13:4803 , piece_id=46HOU2RVDUBAANSTJTDVIXPEVKDTZK3EMP5W2N3QSGXRM3A6J6MQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.226309259 (559h13m45.097392294s) [noise=true , address=77.90.31.14:4662 , piece_id=Y44AXTZHBOSKVA5EJ53JS3C23UHQR7ZCSLYQ62Y42ZHWDWZ2QRQA, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.259740437 (559h13m45.130823472s) [noise=true , address=45.91.116.7:4010 , piece_id=MWQ434WLSAQD2HMRNKHDKF3CXX5VEMQYBB2M6WRHCKQUPKKJBOHQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.275536778 (559h13m45.146619813s) [noise=true , address=77.90.31.18:4808 , piece_id=GZ5NU4MQJBP4RFEIRUVCOXBH4NRQFBADLINHX5BOR6IDYSUYKJXQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.291544058 (559h13m45.162627103s) [noise=true , address=77.90.31.10:4517 , piece_id=ULOEZUXSRPMT27O3XAQN7ZQYNEC76EOSC5AWRK4JBCUCY2AOYW4Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.327271488 (559h13m45.198354563s) [noise=true , address=77.90.31.13:4627 , piece_id=LVJZDX5L4CD6CJW3TKNHICCHNVQRWXRD5TX7VA6NGC64FRHHKO7Q, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.368718174 (559h13m45.239801209s) [noise=true , address=45.91.116.12:4226 , piece_id=WG5HYFUZSA6H376ZQ52NNIUE3YEEHA4XC5HQTFA2PF2T7B7SSOJA, seg_pos={0 0}, upload=490273]: Done uploading piece. err: <nil> Dec 28 22:42:45.368738173 (559h13m45.239821208s) [seg_pos={0 0}, upload=490273]: Segment reached optimal threshold of 13 pieces. Dec 28 22:42:45.368983550 (559h13m45.240066585s) [noise=true , address=45.91.116.10:4083 , piece_id=62OTEKHKQEKAUR6YG546TJJUIQGCD3RKXBICAUINYBQSH25ZZOSQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 28 22:42:45.368990062 (559h13m45.240073097s) [noise=true , address=45.91.116.10:4083 , piece_id=62OTEKHKQEKAUR6YG546TJJUIQGCD3RKXBICAUINYBQSH25ZZOSQ, seg_pos={0 0}, upload=490273]: Long tail cancellation due to optimal threshold Dec 28 22:42:45.369209349 (559h13m45.240292385s) [noise=true , address=45.91.116.6:4974 , piece_id=GABSOBYQXXVDZM7Q67UWTNC2INPS2DU5IMB2G6UM7TKUZHBQYXNQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:42:45.369214579 (559h13m45.240297614s) [noise=true , address=45.91.116.6:4974 , piece_id=GABSOBYQXXVDZM7Q67UWTNC2INPS2DU5IMB2G6UM7TKUZHBQYXNQ, seg_pos={0 0}, upload=490273]: Long tail cancellation due to optimal threshold Dec 28 22:42:45.369314249 (559h13m45.240397284s) [noise=true , address=45.91.116.14:4190 , piece_id=YFMKA36RR5HQUYIFE3MP7FKX453KCB5QBLIIH7ZTV2MK4GCILTQQ, seg_pos={0 0}, upload=490273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 22:42:45.369321673 (559h13m45.240404708s) [noise=true , address=45.91.116.14:4190 , piece_id=YFMKA36RR5HQUYIFE3MP7FKX453KCB5QBLIIH7ZTV2MK4GCILTQQ, seg_pos={0 0}, upload=490273]: Long tail cancellation due to optimal threshold Dec 28 22:42:45.369337073 (559h13m45.240420108s) [seg_pos={0 0}, upload=490273]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:42:45.369401304 (559h13m45.24048434s) [upload=490273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:42:45.369406916 (559h13m45.240489951s) [upload=490273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:42:45.661510754 (559h13m45.532593789s) [upload=490273]: Done uploading segments... Dec 28 22:42:45.661549378 (559h13m45.532632423s) [upload=490273]: Upload finished. err: <nil> Duration: 30.00374338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.X...................X........] duration: 17.220874308s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:42:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 17.220899536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................X....................X.....X....X.X...] duration: 7.746848077s trace: https://jaeger.dev.storj.io/trace/4a2178c2c55bac32 FAIL (2025-12-28 22:42:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 22:39:45.177689502 (559h10m45.048772547s) [upload=490228]: Starting upload Dec 28 22:39:45.177751951 (559h10m45.048834986s) [upload=490228]: Done starting upload Dec 28 22:39:45.178028538 (559h10m45.049111583s) [upload=490228]: Uploading segments... Dec 28 22:39:45.178064927 (559h10m45.049147972s) [upload=490228]: Got next segment. Inline: false Dec 28 22:39:45.178089363 (559h10m45.049172398s) [upload=490228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:39:45.178094452 (559h10m45.049177487s) [upload=490228]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:39:45.427695015 (559h10m45.29877805s) [seg_pos={0 0}, upload=490228]: Begin upload segment... Dec 28 22:39:45.427751452 (559h10m45.298834487s) [seg_pos={0 0}, upload=490228]: Done begin upload segment. Dec 28 22:39:45.427802249 (559h10m45.298885284s) [noise=true , address=145.79.227.32:4836 , piece_id=FEVQNAKSDGQG3OJWFU4VZZVJ5D66BPN6DQZQ4GFXEIPYHLRISAEA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.427885297 (559h10m45.298968332s) [noise=true , address=145.79.227.37:4566 , piece_id=INLOZMG5EA4Z4SRNVTZNUBPFJR3TMZOGXSGCBGXAMZ4HLOCGZWIA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.428344872 (559h10m45.299427907s) [noise=true , address=145.79.227.43:4159 , piece_id=5AOTN3XICHAI5Q7VDQTWLVQA7VJDOZLEALQQXA4VNPUH4LVISMXQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.428693686 (559h10m45.299776721s) [noise=true , address=45.91.116.5:4299 , piece_id=ZY5ERZRRWMCY7DAMG6PO3S67PD7JKPPOCPFZN2UNW4TCSNJHIVMQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.429160103 (559h10m45.300243149s) [noise=true , address=45.91.116.8:4048 , piece_id=JVZWBTNAKQO7K3G7Z2OQTVTRNBKXHGUFITVONKNSWUMHMYSRHSUA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.429558411 (559h10m45.300641447s) [noise=true , address=145.79.227.36:4519 , piece_id=BHVH22VC4LSNANSZ7PEPQGHWGVA742W6AGFS3W7NMHADED5LIATQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.429890574 (559h10m45.300973609s) [noise=true , address=77.90.31.10:4518 , piece_id=Q6OG7WFXH7UKZUH7SCJCOWQBUN72EBZ6KNCMNDESHGH5VDZRF5QQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.430167631 (559h10m45.301250666s) [noise=true , address=145.79.227.2:4115 , piece_id=YE5LWX3DAM6UMNUFWHXN2M6N27JUJZVLWBCSTB5HLLU5YHJROZ6Q, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.430501768 (559h10m45.301584803s) [noise=true , address=145.79.227.19:4079 , piece_id=URLQEPZ77TYVKQLV2K3HCPQLDELMAMRABOB4VJOZFHHR27TFEG3A, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.430609924 (559h10m45.301692959s) [noise=true , address=145.79.227.12:4767 , piece_id=7OZGCTNQ5ANEK2FFJIFZ4ISCNCF5KBPXCLQWSAPLBWOZJELMA6TQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.430907600 (559h10m45.301990675s) [noise=true , address=45.91.116.7:4011 , piece_id=G5MDPLROV6WULA2TAJHMQFE2ZDNTZODYBVVEEZQW5FIFP4W3XNPQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.431102511 (559h10m45.302185546s) [noise=true , address=145.79.227.26:4653 , piece_id=Q5P5EW5BQ4W27BEY7Z6VTXBIUUZNRRO7HEXVPRU7CDZA4QBZK5HA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.431275962 (559h10m45.302358997s) [noise=true , address=145.79.227.16:4909 , piece_id=7RSO6SWF2KDPVNFSX6JWT6XH5M77QWNHKY4ZYJ5ERJYKEB5N2JQA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.431495159 (559h10m45.302578194s) [noise=true , address=145.79.227.38:4978 , piece_id=G4CFAWGMPKXHPHGIYUVUSXGENL7VQVY3UL4AKWUTY2REZLRAXHBA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.431812192 (559h10m45.302895227s) [noise=true , address=145.79.227.14:4837 , piece_id=ZHC7MTMKAZAWTXLHYJ3BMSJFR6BOZNWZFARIEIZMHXVXVTZAUUWQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.432290132 (559h10m45.303373167s) [noise=true , address=145.79.227.18:4006 , piece_id=B3MHNDIEK2HW4V5PTBD2KQOQVKHM2JFWFPDLLSSYQB6K3AKQAUFA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:47.038509332 (559h10m46.909592367s) [upload=490228]: Next returned nil segment Dec 28 22:39:47.038539298 (559h10m46.909622343s) [upload=490228]: Waiting for error group managing segments... Dec 28 22:39:47.419059245 (559h10m47.29014228s) [noise=true , address=145.79.227.19:4079 , piece_id=URLQEPZ77TYVKQLV2K3HCPQLDELMAMRABOB4VJOZFHHR27TFEG3A, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.422934041 (559h10m47.294017076s) [noise=true , address=145.79.227.12:4767 , piece_id=7OZGCTNQ5ANEK2FFJIFZ4ISCNCF5KBPXCLQWSAPLBWOZJELMA6TQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.424723377 (559h10m47.295806412s) [noise=true , address=145.79.227.36:4519 , piece_id=BHVH22VC4LSNANSZ7PEPQGHWGVA742W6AGFS3W7NMHADED5LIATQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.433694584 (559h10m47.304777619s) [noise=true , address=145.79.227.16:4909 , piece_id=7RSO6SWF2KDPVNFSX6JWT6XH5M77QWNHKY4ZYJ5ERJYKEB5N2JQA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.440971206 (559h10m47.312054241s) [noise=true , address=145.79.227.32:4836 , piece_id=FEVQNAKSDGQG3OJWFU4VZZVJ5D66BPN6DQZQ4GFXEIPYHLRISAEA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.442324411 (559h10m47.313407457s) [noise=true , address=145.79.227.37:4566 , piece_id=INLOZMG5EA4Z4SRNVTZNUBPFJR3TMZOGXSGCBGXAMZ4HLOCGZWIA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.448196820 (559h10m47.319279855s) [noise=true , address=145.79.227.26:4653 , piece_id=Q5P5EW5BQ4W27BEY7Z6VTXBIUUZNRRO7HEXVPRU7CDZA4QBZK5HA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.448256935 (559h10m47.31933997s) [noise=true , address=145.79.227.2:4115 , piece_id=YE5LWX3DAM6UMNUFWHXN2M6N27JUJZVLWBCSTB5HLLU5YHJROZ6Q, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.455931443 (559h10m47.327014488s) [noise=true , address=145.79.227.14:4837 , piece_id=ZHC7MTMKAZAWTXLHYJ3BMSJFR6BOZNWZFARIEIZMHXVXVTZAUUWQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.458051369 (559h10m47.329134404s) [noise=true , address=145.79.227.18:4006 , piece_id=B3MHNDIEK2HW4V5PTBD2KQOQVKHM2JFWFPDLLSSYQB6K3AKQAUFA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.459758899 (559h10m47.330841934s) [noise=true , address=145.79.227.38:4978 , piece_id=G4CFAWGMPKXHPHGIYUVUSXGENL7VQVY3UL4AKWUTY2REZLRAXHBA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.637840670 (559h10m47.508923705s) [noise=true , address=77.90.31.10:4518 , piece_id=Q6OG7WFXH7UKZUH7SCJCOWQBUN72EBZ6KNCMNDESHGH5VDZRF5QQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.710012815 (559h10m47.58109587s) [noise=true , address=45.91.116.8:4048 , piece_id=JVZWBTNAKQO7K3G7Z2OQTVTRNBKXHGUFITVONKNSWUMHMYSRHSUA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.710036048 (559h10m47.581119083s) [seg_pos={0 0}, upload=490228]: Segment reached optimal threshold of 13 pieces. Dec 28 22:39:47.710222603 (559h10m47.581305648s) [noise=true , address=45.91.116.7:4011 , piece_id=G5MDPLROV6WULA2TAJHMQFE2ZDNTZODYBVVEEZQW5FIFP4W3XNPQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:58128->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:58128->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58128->45.91.116.7:4011: use of closed network connection Dec 28 22:39:47.710229056 (559h10m47.581312091s) [noise=true , address=45.91.116.7:4011 , piece_id=G5MDPLROV6WULA2TAJHMQFE2ZDNTZODYBVVEEZQW5FIFP4W3XNPQ, seg_pos={0 0}, upload=490228]: Long tail cancellation due to optimal threshold Dec 28 22:39:47.710338515 (559h10m47.58142155s) [noise=true , address=45.91.116.5:4299 , piece_id=ZY5ERZRRWMCY7DAMG6PO3S67PD7JKPPOCPFZN2UNW4TCSNJHIVMQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 28 22:39:47.710344215 (559h10m47.58142725s) [noise=true , address=45.91.116.5:4299 , piece_id=ZY5ERZRRWMCY7DAMG6PO3S67PD7JKPPOCPFZN2UNW4TCSNJHIVMQ, seg_pos={0 0}, upload=490228]: Long tail cancellation due to optimal threshold Dec 28 22:39:47.710654887 (559h10m47.581737932s) [noise=true , address=145.79.227.43:4159 , piece_id=5AOTN3XICHAI5Q7VDQTWLVQA7VJDOZLEALQQXA4VNPUH4LVISMXQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 22:39:47.710673572 (559h10m47.581756607s) [noise=true , address=145.79.227.43:4159 , piece_id=5AOTN3XICHAI5Q7VDQTWLVQA7VJDOZLEALQQXA4VNPUH4LVISMXQ, seg_pos={0 0}, upload=490228]: Long tail cancellation due to optimal threshold Dec 28 22:39:47.710700403 (559h10m47.581783438s) [seg_pos={0 0}, upload=490228]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:39:47.710870558 (559h10m47.581953603s) [upload=490228]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:39:47.710883772 (559h10m47.581966827s) [upload=490228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:39:48.000462766 (559h10m47.871545831s) [upload=490228]: Done uploading segments... Dec 28 22:39:48.000597333 (559h10m47.871680388s) [upload=490228]: Upload finished. err: <nil> Duration: 7.746874276s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.X...................X.......] duration: 31.511434183s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:42:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 31.511458348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................X....................X.....X....X.X..] duration: 5.809073835s trace: https://jaeger.dev.storj.io/trace/4a2178c2c55bac32 FAIL (2025-12-28 22:41:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 22:39:45.177689502 (559h10m45.048772547s) [upload=490228]: Starting upload Dec 28 22:39:45.177751951 (559h10m45.048834986s) [upload=490228]: Done starting upload Dec 28 22:39:45.178028538 (559h10m45.049111583s) [upload=490228]: Uploading segments... Dec 28 22:39:45.178064927 (559h10m45.049147972s) [upload=490228]: Got next segment. Inline: false Dec 28 22:39:45.178089363 (559h10m45.049172398s) [upload=490228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:39:45.178094452 (559h10m45.049177487s) [upload=490228]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:39:45.427695015 (559h10m45.29877805s) [seg_pos={0 0}, upload=490228]: Begin upload segment... Dec 28 22:39:45.427751452 (559h10m45.298834487s) [seg_pos={0 0}, upload=490228]: Done begin upload segment. Dec 28 22:39:45.427802249 (559h10m45.298885284s) [noise=true , address=145.79.227.32:4836 , piece_id=FEVQNAKSDGQG3OJWFU4VZZVJ5D66BPN6DQZQ4GFXEIPYHLRISAEA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.427885297 (559h10m45.298968332s) [noise=true , address=145.79.227.37:4566 , piece_id=INLOZMG5EA4Z4SRNVTZNUBPFJR3TMZOGXSGCBGXAMZ4HLOCGZWIA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.428344872 (559h10m45.299427907s) [noise=true , address=145.79.227.43:4159 , piece_id=5AOTN3XICHAI5Q7VDQTWLVQA7VJDOZLEALQQXA4VNPUH4LVISMXQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.428693686 (559h10m45.299776721s) [noise=true , address=45.91.116.5:4299 , piece_id=ZY5ERZRRWMCY7DAMG6PO3S67PD7JKPPOCPFZN2UNW4TCSNJHIVMQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.429160103 (559h10m45.300243149s) [noise=true , address=45.91.116.8:4048 , piece_id=JVZWBTNAKQO7K3G7Z2OQTVTRNBKXHGUFITVONKNSWUMHMYSRHSUA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.429558411 (559h10m45.300641447s) [noise=true , address=145.79.227.36:4519 , piece_id=BHVH22VC4LSNANSZ7PEPQGHWGVA742W6AGFS3W7NMHADED5LIATQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.429890574 (559h10m45.300973609s) [noise=true , address=77.90.31.10:4518 , piece_id=Q6OG7WFXH7UKZUH7SCJCOWQBUN72EBZ6KNCMNDESHGH5VDZRF5QQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.430167631 (559h10m45.301250666s) [noise=true , address=145.79.227.2:4115 , piece_id=YE5LWX3DAM6UMNUFWHXN2M6N27JUJZVLWBCSTB5HLLU5YHJROZ6Q, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.430501768 (559h10m45.301584803s) [noise=true , address=145.79.227.19:4079 , piece_id=URLQEPZ77TYVKQLV2K3HCPQLDELMAMRABOB4VJOZFHHR27TFEG3A, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.430609924 (559h10m45.301692959s) [noise=true , address=145.79.227.12:4767 , piece_id=7OZGCTNQ5ANEK2FFJIFZ4ISCNCF5KBPXCLQWSAPLBWOZJELMA6TQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.430907600 (559h10m45.301990675s) [noise=true , address=45.91.116.7:4011 , piece_id=G5MDPLROV6WULA2TAJHMQFE2ZDNTZODYBVVEEZQW5FIFP4W3XNPQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.431102511 (559h10m45.302185546s) [noise=true , address=145.79.227.26:4653 , piece_id=Q5P5EW5BQ4W27BEY7Z6VTXBIUUZNRRO7HEXVPRU7CDZA4QBZK5HA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.431275962 (559h10m45.302358997s) [noise=true , address=145.79.227.16:4909 , piece_id=7RSO6SWF2KDPVNFSX6JWT6XH5M77QWNHKY4ZYJ5ERJYKEB5N2JQA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.431495159 (559h10m45.302578194s) [noise=true , address=145.79.227.38:4978 , piece_id=G4CFAWGMPKXHPHGIYUVUSXGENL7VQVY3UL4AKWUTY2REZLRAXHBA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.431812192 (559h10m45.302895227s) [noise=true , address=145.79.227.14:4837 , piece_id=ZHC7MTMKAZAWTXLHYJ3BMSJFR6BOZNWZFARIEIZMHXVXVTZAUUWQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.432290132 (559h10m45.303373167s) [noise=true , address=145.79.227.18:4006 , piece_id=B3MHNDIEK2HW4V5PTBD2KQOQVKHM2JFWFPDLLSSYQB6K3AKQAUFA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:47.038509332 (559h10m46.909592367s) [upload=490228]: Next returned nil segment Dec 28 22:39:47.038539298 (559h10m46.909622343s) [upload=490228]: Waiting for error group managing segments... Dec 28 22:39:47.419059245 (559h10m47.29014228s) [noise=true , address=145.79.227.19:4079 , piece_id=URLQEPZ77TYVKQLV2K3HCPQLDELMAMRABOB4VJOZFHHR27TFEG3A, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.422934041 (559h10m47.294017076s) [noise=true , address=145.79.227.12:4767 , piece_id=7OZGCTNQ5ANEK2FFJIFZ4ISCNCF5KBPXCLQWSAPLBWOZJELMA6TQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.424723377 (559h10m47.295806412s) [noise=true , address=145.79.227.36:4519 , piece_id=BHVH22VC4LSNANSZ7PEPQGHWGVA742W6AGFS3W7NMHADED5LIATQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.433694584 (559h10m47.304777619s) [noise=true , address=145.79.227.16:4909 , piece_id=7RSO6SWF2KDPVNFSX6JWT6XH5M77QWNHKY4ZYJ5ERJYKEB5N2JQA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.440971206 (559h10m47.312054241s) [noise=true , address=145.79.227.32:4836 , piece_id=FEVQNAKSDGQG3OJWFU4VZZVJ5D66BPN6DQZQ4GFXEIPYHLRISAEA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.442324411 (559h10m47.313407457s) [noise=true , address=145.79.227.37:4566 , piece_id=INLOZMG5EA4Z4SRNVTZNUBPFJR3TMZOGXSGCBGXAMZ4HLOCGZWIA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.448196820 (559h10m47.319279855s) [noise=true , address=145.79.227.26:4653 , piece_id=Q5P5EW5BQ4W27BEY7Z6VTXBIUUZNRRO7HEXVPRU7CDZA4QBZK5HA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.448256935 (559h10m47.31933997s) [noise=true , address=145.79.227.2:4115 , piece_id=YE5LWX3DAM6UMNUFWHXN2M6N27JUJZVLWBCSTB5HLLU5YHJROZ6Q, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.455931443 (559h10m47.327014488s) [noise=true , address=145.79.227.14:4837 , piece_id=ZHC7MTMKAZAWTXLHYJ3BMSJFR6BOZNWZFARIEIZMHXVXVTZAUUWQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.458051369 (559h10m47.329134404s) [noise=true , address=145.79.227.18:4006 , piece_id=B3MHNDIEK2HW4V5PTBD2KQOQVKHM2JFWFPDLLSSYQB6K3AKQAUFA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.459758899 (559h10m47.330841934s) [noise=true , address=145.79.227.38:4978 , piece_id=G4CFAWGMPKXHPHGIYUVUSXGENL7VQVY3UL4AKWUTY2REZLRAXHBA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.637840670 (559h10m47.508923705s) [noise=true , address=77.90.31.10:4518 , piece_id=Q6OG7WFXH7UKZUH7SCJCOWQBUN72EBZ6KNCMNDESHGH5VDZRF5QQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.710012815 (559h10m47.58109587s) [noise=true , address=45.91.116.8:4048 , piece_id=JVZWBTNAKQO7K3G7Z2OQTVTRNBKXHGUFITVONKNSWUMHMYSRHSUA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.710036048 (559h10m47.581119083s) [seg_pos={0 0}, upload=490228]: Segment reached optimal threshold of 13 pieces. Dec 28 22:39:47.710222603 (559h10m47.581305648s) [noise=true , address=45.91.116.7:4011 , piece_id=G5MDPLROV6WULA2TAJHMQFE2ZDNTZODYBVVEEZQW5FIFP4W3XNPQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:58128->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:58128->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58128->45.91.116.7:4011: use of closed network connection Dec 28 22:39:47.710229056 (559h10m47.581312091s) [noise=true , address=45.91.116.7:4011 , piece_id=G5MDPLROV6WULA2TAJHMQFE2ZDNTZODYBVVEEZQW5FIFP4W3XNPQ, seg_pos={0 0}, upload=490228]: Long tail cancellation due to optimal threshold Dec 28 22:39:47.710338515 (559h10m47.58142155s) [noise=true , address=45.91.116.5:4299 , piece_id=ZY5ERZRRWMCY7DAMG6PO3S67PD7JKPPOCPFZN2UNW4TCSNJHIVMQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 28 22:39:47.710344215 (559h10m47.58142725s) [noise=true , address=45.91.116.5:4299 , piece_id=ZY5ERZRRWMCY7DAMG6PO3S67PD7JKPPOCPFZN2UNW4TCSNJHIVMQ, seg_pos={0 0}, upload=490228]: Long tail cancellation due to optimal threshold Dec 28 22:39:47.710654887 (559h10m47.581737932s) [noise=true , address=145.79.227.43:4159 , piece_id=5AOTN3XICHAI5Q7VDQTWLVQA7VJDOZLEALQQXA4VNPUH4LVISMXQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 22:39:47.710673572 (559h10m47.581756607s) [noise=true , address=145.79.227.43:4159 , piece_id=5AOTN3XICHAI5Q7VDQTWLVQA7VJDOZLEALQQXA4VNPUH4LVISMXQ, seg_pos={0 0}, upload=490228]: Long tail cancellation due to optimal threshold Dec 28 22:39:47.710700403 (559h10m47.581783438s) [seg_pos={0 0}, upload=490228]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:39:47.710870558 (559h10m47.581953603s) [upload=490228]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:39:47.710883772 (559h10m47.581966827s) [upload=490228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:39:48.000462766 (559h10m47.871545831s) [upload=490228]: Done uploading segments... Dec 28 22:39:48.000597333 (559h10m47.871680388s) [upload=490228]: Upload finished. err: <nil> Duration: 5.809099764s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.X...................X......] duration: 15.145255186s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:40:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 15.145285856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X.X...................X.....] duration: 16.29076416s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:40:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 16.290793516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................X....................X.....X....X.X.] duration: 6.110959793s trace: https://jaeger.dev.storj.io/trace/4a2178c2c55bac32 FAIL (2025-12-28 22:40:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 22:39:45.177689502 (559h10m45.048772547s) [upload=490228]: Starting upload Dec 28 22:39:45.177751951 (559h10m45.048834986s) [upload=490228]: Done starting upload Dec 28 22:39:45.178028538 (559h10m45.049111583s) [upload=490228]: Uploading segments... Dec 28 22:39:45.178064927 (559h10m45.049147972s) [upload=490228]: Got next segment. Inline: false Dec 28 22:39:45.178089363 (559h10m45.049172398s) [upload=490228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:39:45.178094452 (559h10m45.049177487s) [upload=490228]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:39:45.427695015 (559h10m45.29877805s) [seg_pos={0 0}, upload=490228]: Begin upload segment... Dec 28 22:39:45.427751452 (559h10m45.298834487s) [seg_pos={0 0}, upload=490228]: Done begin upload segment. Dec 28 22:39:45.427802249 (559h10m45.298885284s) [noise=true , address=145.79.227.32:4836 , piece_id=FEVQNAKSDGQG3OJWFU4VZZVJ5D66BPN6DQZQ4GFXEIPYHLRISAEA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.427885297 (559h10m45.298968332s) [noise=true , address=145.79.227.37:4566 , piece_id=INLOZMG5EA4Z4SRNVTZNUBPFJR3TMZOGXSGCBGXAMZ4HLOCGZWIA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.428344872 (559h10m45.299427907s) [noise=true , address=145.79.227.43:4159 , piece_id=5AOTN3XICHAI5Q7VDQTWLVQA7VJDOZLEALQQXA4VNPUH4LVISMXQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.428693686 (559h10m45.299776721s) [noise=true , address=45.91.116.5:4299 , piece_id=ZY5ERZRRWMCY7DAMG6PO3S67PD7JKPPOCPFZN2UNW4TCSNJHIVMQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.429160103 (559h10m45.300243149s) [noise=true , address=45.91.116.8:4048 , piece_id=JVZWBTNAKQO7K3G7Z2OQTVTRNBKXHGUFITVONKNSWUMHMYSRHSUA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.429558411 (559h10m45.300641447s) [noise=true , address=145.79.227.36:4519 , piece_id=BHVH22VC4LSNANSZ7PEPQGHWGVA742W6AGFS3W7NMHADED5LIATQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.429890574 (559h10m45.300973609s) [noise=true , address=77.90.31.10:4518 , piece_id=Q6OG7WFXH7UKZUH7SCJCOWQBUN72EBZ6KNCMNDESHGH5VDZRF5QQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.430167631 (559h10m45.301250666s) [noise=true , address=145.79.227.2:4115 , piece_id=YE5LWX3DAM6UMNUFWHXN2M6N27JUJZVLWBCSTB5HLLU5YHJROZ6Q, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.430501768 (559h10m45.301584803s) [noise=true , address=145.79.227.19:4079 , piece_id=URLQEPZ77TYVKQLV2K3HCPQLDELMAMRABOB4VJOZFHHR27TFEG3A, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.430609924 (559h10m45.301692959s) [noise=true , address=145.79.227.12:4767 , piece_id=7OZGCTNQ5ANEK2FFJIFZ4ISCNCF5KBPXCLQWSAPLBWOZJELMA6TQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.430907600 (559h10m45.301990675s) [noise=true , address=45.91.116.7:4011 , piece_id=G5MDPLROV6WULA2TAJHMQFE2ZDNTZODYBVVEEZQW5FIFP4W3XNPQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.431102511 (559h10m45.302185546s) [noise=true , address=145.79.227.26:4653 , piece_id=Q5P5EW5BQ4W27BEY7Z6VTXBIUUZNRRO7HEXVPRU7CDZA4QBZK5HA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.431275962 (559h10m45.302358997s) [noise=true , address=145.79.227.16:4909 , piece_id=7RSO6SWF2KDPVNFSX6JWT6XH5M77QWNHKY4ZYJ5ERJYKEB5N2JQA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.431495159 (559h10m45.302578194s) [noise=true , address=145.79.227.38:4978 , piece_id=G4CFAWGMPKXHPHGIYUVUSXGENL7VQVY3UL4AKWUTY2REZLRAXHBA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.431812192 (559h10m45.302895227s) [noise=true , address=145.79.227.14:4837 , piece_id=ZHC7MTMKAZAWTXLHYJ3BMSJFR6BOZNWZFARIEIZMHXVXVTZAUUWQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.432290132 (559h10m45.303373167s) [noise=true , address=145.79.227.18:4006 , piece_id=B3MHNDIEK2HW4V5PTBD2KQOQVKHM2JFWFPDLLSSYQB6K3AKQAUFA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:47.038509332 (559h10m46.909592367s) [upload=490228]: Next returned nil segment Dec 28 22:39:47.038539298 (559h10m46.909622343s) [upload=490228]: Waiting for error group managing segments... Dec 28 22:39:47.419059245 (559h10m47.29014228s) [noise=true , address=145.79.227.19:4079 , piece_id=URLQEPZ77TYVKQLV2K3HCPQLDELMAMRABOB4VJOZFHHR27TFEG3A, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.422934041 (559h10m47.294017076s) [noise=true , address=145.79.227.12:4767 , piece_id=7OZGCTNQ5ANEK2FFJIFZ4ISCNCF5KBPXCLQWSAPLBWOZJELMA6TQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.424723377 (559h10m47.295806412s) [noise=true , address=145.79.227.36:4519 , piece_id=BHVH22VC4LSNANSZ7PEPQGHWGVA742W6AGFS3W7NMHADED5LIATQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.433694584 (559h10m47.304777619s) [noise=true , address=145.79.227.16:4909 , piece_id=7RSO6SWF2KDPVNFSX6JWT6XH5M77QWNHKY4ZYJ5ERJYKEB5N2JQA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.440971206 (559h10m47.312054241s) [noise=true , address=145.79.227.32:4836 , piece_id=FEVQNAKSDGQG3OJWFU4VZZVJ5D66BPN6DQZQ4GFXEIPYHLRISAEA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.442324411 (559h10m47.313407457s) [noise=true , address=145.79.227.37:4566 , piece_id=INLOZMG5EA4Z4SRNVTZNUBPFJR3TMZOGXSGCBGXAMZ4HLOCGZWIA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.448196820 (559h10m47.319279855s) [noise=true , address=145.79.227.26:4653 , piece_id=Q5P5EW5BQ4W27BEY7Z6VTXBIUUZNRRO7HEXVPRU7CDZA4QBZK5HA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.448256935 (559h10m47.31933997s) [noise=true , address=145.79.227.2:4115 , piece_id=YE5LWX3DAM6UMNUFWHXN2M6N27JUJZVLWBCSTB5HLLU5YHJROZ6Q, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.455931443 (559h10m47.327014488s) [noise=true , address=145.79.227.14:4837 , piece_id=ZHC7MTMKAZAWTXLHYJ3BMSJFR6BOZNWZFARIEIZMHXVXVTZAUUWQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.458051369 (559h10m47.329134404s) [noise=true , address=145.79.227.18:4006 , piece_id=B3MHNDIEK2HW4V5PTBD2KQOQVKHM2JFWFPDLLSSYQB6K3AKQAUFA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.459758899 (559h10m47.330841934s) [noise=true , address=145.79.227.38:4978 , piece_id=G4CFAWGMPKXHPHGIYUVUSXGENL7VQVY3UL4AKWUTY2REZLRAXHBA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.637840670 (559h10m47.508923705s) [noise=true , address=77.90.31.10:4518 , piece_id=Q6OG7WFXH7UKZUH7SCJCOWQBUN72EBZ6KNCMNDESHGH5VDZRF5QQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.710012815 (559h10m47.58109587s) [noise=true , address=45.91.116.8:4048 , piece_id=JVZWBTNAKQO7K3G7Z2OQTVTRNBKXHGUFITVONKNSWUMHMYSRHSUA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.710036048 (559h10m47.581119083s) [seg_pos={0 0}, upload=490228]: Segment reached optimal threshold of 13 pieces. Dec 28 22:39:47.710222603 (559h10m47.581305648s) [noise=true , address=45.91.116.7:4011 , piece_id=G5MDPLROV6WULA2TAJHMQFE2ZDNTZODYBVVEEZQW5FIFP4W3XNPQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:58128->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:58128->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58128->45.91.116.7:4011: use of closed network connection Dec 28 22:39:47.710229056 (559h10m47.581312091s) [noise=true , address=45.91.116.7:4011 , piece_id=G5MDPLROV6WULA2TAJHMQFE2ZDNTZODYBVVEEZQW5FIFP4W3XNPQ, seg_pos={0 0}, upload=490228]: Long tail cancellation due to optimal threshold Dec 28 22:39:47.710338515 (559h10m47.58142155s) [noise=true , address=45.91.116.5:4299 , piece_id=ZY5ERZRRWMCY7DAMG6PO3S67PD7JKPPOCPFZN2UNW4TCSNJHIVMQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 28 22:39:47.710344215 (559h10m47.58142725s) [noise=true , address=45.91.116.5:4299 , piece_id=ZY5ERZRRWMCY7DAMG6PO3S67PD7JKPPOCPFZN2UNW4TCSNJHIVMQ, seg_pos={0 0}, upload=490228]: Long tail cancellation due to optimal threshold Dec 28 22:39:47.710654887 (559h10m47.581737932s) [noise=true , address=145.79.227.43:4159 , piece_id=5AOTN3XICHAI5Q7VDQTWLVQA7VJDOZLEALQQXA4VNPUH4LVISMXQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 22:39:47.710673572 (559h10m47.581756607s) [noise=true , address=145.79.227.43:4159 , piece_id=5AOTN3XICHAI5Q7VDQTWLVQA7VJDOZLEALQQXA4VNPUH4LVISMXQ, seg_pos={0 0}, upload=490228]: Long tail cancellation due to optimal threshold Dec 28 22:39:47.710700403 (559h10m47.581783438s) [seg_pos={0 0}, upload=490228]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:39:47.710870558 (559h10m47.581953603s) [upload=490228]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:39:47.710883772 (559h10m47.581966827s) [upload=490228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:39:48.000462766 (559h10m47.871545831s) [upload=490228]: Done uploading segments... Dec 28 22:39:48.000597333 (559h10m47.871680388s) [upload=490228]: Upload finished. err: <nil> Duration: 6.110976274s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................X....................X.....X....X.X] duration: 30.002550922s trace: https://jaeger.dev.storj.io/trace/4a2178c2c55bac32 FAIL (2025-12-28 22:40:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 22:39:45.177689502 (559h10m45.048772547s) [upload=490228]: Starting upload Dec 28 22:39:45.177751951 (559h10m45.048834986s) [upload=490228]: Done starting upload Dec 28 22:39:45.178028538 (559h10m45.049111583s) [upload=490228]: Uploading segments... Dec 28 22:39:45.178064927 (559h10m45.049147972s) [upload=490228]: Got next segment. Inline: false Dec 28 22:39:45.178089363 (559h10m45.049172398s) [upload=490228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:39:45.178094452 (559h10m45.049177487s) [upload=490228]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:39:45.427695015 (559h10m45.29877805s) [seg_pos={0 0}, upload=490228]: Begin upload segment... Dec 28 22:39:45.427751452 (559h10m45.298834487s) [seg_pos={0 0}, upload=490228]: Done begin upload segment. Dec 28 22:39:45.427802249 (559h10m45.298885284s) [noise=true , address=145.79.227.32:4836 , piece_id=FEVQNAKSDGQG3OJWFU4VZZVJ5D66BPN6DQZQ4GFXEIPYHLRISAEA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.427885297 (559h10m45.298968332s) [noise=true , address=145.79.227.37:4566 , piece_id=INLOZMG5EA4Z4SRNVTZNUBPFJR3TMZOGXSGCBGXAMZ4HLOCGZWIA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.428344872 (559h10m45.299427907s) [noise=true , address=145.79.227.43:4159 , piece_id=5AOTN3XICHAI5Q7VDQTWLVQA7VJDOZLEALQQXA4VNPUH4LVISMXQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.428693686 (559h10m45.299776721s) [noise=true , address=45.91.116.5:4299 , piece_id=ZY5ERZRRWMCY7DAMG6PO3S67PD7JKPPOCPFZN2UNW4TCSNJHIVMQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.429160103 (559h10m45.300243149s) [noise=true , address=45.91.116.8:4048 , piece_id=JVZWBTNAKQO7K3G7Z2OQTVTRNBKXHGUFITVONKNSWUMHMYSRHSUA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.429558411 (559h10m45.300641447s) [noise=true , address=145.79.227.36:4519 , piece_id=BHVH22VC4LSNANSZ7PEPQGHWGVA742W6AGFS3W7NMHADED5LIATQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.429890574 (559h10m45.300973609s) [noise=true , address=77.90.31.10:4518 , piece_id=Q6OG7WFXH7UKZUH7SCJCOWQBUN72EBZ6KNCMNDESHGH5VDZRF5QQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.430167631 (559h10m45.301250666s) [noise=true , address=145.79.227.2:4115 , piece_id=YE5LWX3DAM6UMNUFWHXN2M6N27JUJZVLWBCSTB5HLLU5YHJROZ6Q, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.430501768 (559h10m45.301584803s) [noise=true , address=145.79.227.19:4079 , piece_id=URLQEPZ77TYVKQLV2K3HCPQLDELMAMRABOB4VJOZFHHR27TFEG3A, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.430609924 (559h10m45.301692959s) [noise=true , address=145.79.227.12:4767 , piece_id=7OZGCTNQ5ANEK2FFJIFZ4ISCNCF5KBPXCLQWSAPLBWOZJELMA6TQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.430907600 (559h10m45.301990675s) [noise=true , address=45.91.116.7:4011 , piece_id=G5MDPLROV6WULA2TAJHMQFE2ZDNTZODYBVVEEZQW5FIFP4W3XNPQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.431102511 (559h10m45.302185546s) [noise=true , address=145.79.227.26:4653 , piece_id=Q5P5EW5BQ4W27BEY7Z6VTXBIUUZNRRO7HEXVPRU7CDZA4QBZK5HA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.431275962 (559h10m45.302358997s) [noise=true , address=145.79.227.16:4909 , piece_id=7RSO6SWF2KDPVNFSX6JWT6XH5M77QWNHKY4ZYJ5ERJYKEB5N2JQA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.431495159 (559h10m45.302578194s) [noise=true , address=145.79.227.38:4978 , piece_id=G4CFAWGMPKXHPHGIYUVUSXGENL7VQVY3UL4AKWUTY2REZLRAXHBA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.431812192 (559h10m45.302895227s) [noise=true , address=145.79.227.14:4837 , piece_id=ZHC7MTMKAZAWTXLHYJ3BMSJFR6BOZNWZFARIEIZMHXVXVTZAUUWQ, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:45.432290132 (559h10m45.303373167s) [noise=true , address=145.79.227.18:4006 , piece_id=B3MHNDIEK2HW4V5PTBD2KQOQVKHM2JFWFPDLLSSYQB6K3AKQAUFA, seg_pos={0 0}, upload=490228]: Uploading piece... Dec 28 22:39:47.038509332 (559h10m46.909592367s) [upload=490228]: Next returned nil segment Dec 28 22:39:47.038539298 (559h10m46.909622343s) [upload=490228]: Waiting for error group managing segments... Dec 28 22:39:47.419059245 (559h10m47.29014228s) [noise=true , address=145.79.227.19:4079 , piece_id=URLQEPZ77TYVKQLV2K3HCPQLDELMAMRABOB4VJOZFHHR27TFEG3A, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.422934041 (559h10m47.294017076s) [noise=true , address=145.79.227.12:4767 , piece_id=7OZGCTNQ5ANEK2FFJIFZ4ISCNCF5KBPXCLQWSAPLBWOZJELMA6TQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.424723377 (559h10m47.295806412s) [noise=true , address=145.79.227.36:4519 , piece_id=BHVH22VC4LSNANSZ7PEPQGHWGVA742W6AGFS3W7NMHADED5LIATQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.433694584 (559h10m47.304777619s) [noise=true , address=145.79.227.16:4909 , piece_id=7RSO6SWF2KDPVNFSX6JWT6XH5M77QWNHKY4ZYJ5ERJYKEB5N2JQA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.440971206 (559h10m47.312054241s) [noise=true , address=145.79.227.32:4836 , piece_id=FEVQNAKSDGQG3OJWFU4VZZVJ5D66BPN6DQZQ4GFXEIPYHLRISAEA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.442324411 (559h10m47.313407457s) [noise=true , address=145.79.227.37:4566 , piece_id=INLOZMG5EA4Z4SRNVTZNUBPFJR3TMZOGXSGCBGXAMZ4HLOCGZWIA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.448196820 (559h10m47.319279855s) [noise=true , address=145.79.227.26:4653 , piece_id=Q5P5EW5BQ4W27BEY7Z6VTXBIUUZNRRO7HEXVPRU7CDZA4QBZK5HA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.448256935 (559h10m47.31933997s) [noise=true , address=145.79.227.2:4115 , piece_id=YE5LWX3DAM6UMNUFWHXN2M6N27JUJZVLWBCSTB5HLLU5YHJROZ6Q, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.455931443 (559h10m47.327014488s) [noise=true , address=145.79.227.14:4837 , piece_id=ZHC7MTMKAZAWTXLHYJ3BMSJFR6BOZNWZFARIEIZMHXVXVTZAUUWQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.458051369 (559h10m47.329134404s) [noise=true , address=145.79.227.18:4006 , piece_id=B3MHNDIEK2HW4V5PTBD2KQOQVKHM2JFWFPDLLSSYQB6K3AKQAUFA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.459758899 (559h10m47.330841934s) [noise=true , address=145.79.227.38:4978 , piece_id=G4CFAWGMPKXHPHGIYUVUSXGENL7VQVY3UL4AKWUTY2REZLRAXHBA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.637840670 (559h10m47.508923705s) [noise=true , address=77.90.31.10:4518 , piece_id=Q6OG7WFXH7UKZUH7SCJCOWQBUN72EBZ6KNCMNDESHGH5VDZRF5QQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.710012815 (559h10m47.58109587s) [noise=true , address=45.91.116.8:4048 , piece_id=JVZWBTNAKQO7K3G7Z2OQTVTRNBKXHGUFITVONKNSWUMHMYSRHSUA, seg_pos={0 0}, upload=490228]: Done uploading piece. err: <nil> Dec 28 22:39:47.710036048 (559h10m47.581119083s) [seg_pos={0 0}, upload=490228]: Segment reached optimal threshold of 13 pieces. Dec 28 22:39:47.710222603 (559h10m47.581305648s) [noise=true , address=45.91.116.7:4011 , piece_id=G5MDPLROV6WULA2TAJHMQFE2ZDNTZODYBVVEEZQW5FIFP4W3XNPQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:58128->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:58128->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58128->45.91.116.7:4011: use of closed network connection Dec 28 22:39:47.710229056 (559h10m47.581312091s) [noise=true , address=45.91.116.7:4011 , piece_id=G5MDPLROV6WULA2TAJHMQFE2ZDNTZODYBVVEEZQW5FIFP4W3XNPQ, seg_pos={0 0}, upload=490228]: Long tail cancellation due to optimal threshold Dec 28 22:39:47.710338515 (559h10m47.58142155s) [noise=true , address=45.91.116.5:4299 , piece_id=ZY5ERZRRWMCY7DAMG6PO3S67PD7JKPPOCPFZN2UNW4TCSNJHIVMQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 28 22:39:47.710344215 (559h10m47.58142725s) [noise=true , address=45.91.116.5:4299 , piece_id=ZY5ERZRRWMCY7DAMG6PO3S67PD7JKPPOCPFZN2UNW4TCSNJHIVMQ, seg_pos={0 0}, upload=490228]: Long tail cancellation due to optimal threshold Dec 28 22:39:47.710654887 (559h10m47.581737932s) [noise=true , address=145.79.227.43:4159 , piece_id=5AOTN3XICHAI5Q7VDQTWLVQA7VJDOZLEALQQXA4VNPUH4LVISMXQ, seg_pos={0 0}, upload=490228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 22:39:47.710673572 (559h10m47.581756607s) [noise=true , address=145.79.227.43:4159 , piece_id=5AOTN3XICHAI5Q7VDQTWLVQA7VJDOZLEALQQXA4VNPUH4LVISMXQ, seg_pos={0 0}, upload=490228]: Long tail cancellation due to optimal threshold Dec 28 22:39:47.710700403 (559h10m47.581783438s) [seg_pos={0 0}, upload=490228]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:39:47.710870558 (559h10m47.581953603s) [upload=490228]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:39:47.710883772 (559h10m47.581966827s) [upload=490228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:39:48.000462766 (559h10m47.871545831s) [upload=490228]: Done uploading segments... Dec 28 22:39:48.000597333 (559h10m47.871680388s) [upload=490228]: Upload finished. err: <nil> Duration: 30.002578385s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.X...................X....] duration: 15.445095306s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:39:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 15.445124933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................X....................X.....X....X.] duration: 5.738176044s trace: https://jaeger.dev.storj.io/trace/5f5562a3131aec10 FAIL (2025-12-28 22:39:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: KG5UVS6GY5A2J7R66MLRVNPW5XORQQWZO2M5FA5FMFAAIQTQ76TQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: BSHPBQ2KDFJ2LHQTNQXZ5GTYUXPDMPIH2Z7DU5PDMVUQWWZLCWHA) context deadline exceeded ======LOGS====== Dec 28 22:37:49.917195004 (559h8m49.788278039s) [upload=490202]: Starting upload Dec 28 22:37:49.917247603 (559h8m49.788330638s) [upload=490202]: Done starting upload Dec 28 22:37:49.917366641 (559h8m49.788449686s) [upload=490202]: Uploading segments... Dec 28 22:37:49.917459407 (559h8m49.788542442s) [upload=490202]: Got next segment. Inline: false Dec 28 22:37:49.917492500 (559h8m49.788575545s) [upload=490202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:37:49.917497270 (559h8m49.788580315s) [upload=490202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:37:50.166914830 (559h8m50.037997865s) [seg_pos={0 0}, upload=490202]: Begin upload segment... Dec 28 22:37:50.166965275 (559h8m50.03804831s) [seg_pos={0 0}, upload=490202]: Done begin upload segment. Dec 28 22:37:50.167009459 (559h8m50.038092494s) [noise=true , address=145.79.227.50:4374 , piece_id=KATXKHW5L3BXASF7MNQU2LOTHZ2GPDBOFL4VGMY4BYNJTPW3LYOQ, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.167066328 (559h8m50.038149393s) [noise=true , address=145.79.227.16:4912 , piece_id=J5CR6HNRR62DKVGL4O2P45CFC4LSKMPARVRZ4D7IU3RRYDCKMU4A, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.167686259 (559h8m50.038769294s) [noise=true , address=145.79.227.21:4473 , piece_id=UL5ANBOVC2E3VFYNUT3H2BNSX45FWGWF73HT3PTQ6FOSJC2GUOAA, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.168137247 (559h8m50.039220282s) [noise=true , address=145.79.227.2:4114 , piece_id=6TJZJMIEIIFKPLU6ANH23JICEKVJ4P3KBPQSSJYUUUWDDDPQWQLQ, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.168565432 (559h8m50.039648477s) [noise=true , address=77.90.31.21:4880 , piece_id=LSKLRPZVVODLA2W4JX2NCOGEVAQKHFNFSNBRJZE3QN2J5MVVNIBA, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.169995636 (559h8m50.041078681s) [noise=true , address=145.79.227.5:4222 , piece_id=L5GHMXVRGRNG7MRAXDQ5JBAKGWIH32SA5QQCU6UHOQ5O3POTXY4A, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.170943240 (559h8m50.042026275s) [noise=true , address=145.79.227.13:4802 , piece_id=7SY56FHEGZ4TZN2SIUKP4I5PG2EMJF4AOS52A2O6ELXRYUTPTLLA, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.171223303 (559h8m50.042306348s) [noise=true , address=145.79.227.4:4187 , piece_id=VDGZC2AVKT3HCMQSFLN4P3EGZJRSUG4VRFFC7UQXXCBHRVJPICBQ, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.172627797 (559h8m50.043710832s) [noise=true , address=45.91.116.12:4226 , piece_id=Z22UGEYHKTKDY5Q4QBWBFUBFEPOWMKWPIZMM323RILMFYBHM5ECA, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.173903606 (559h8m50.044986641s) [noise=true , address=145.79.227.52:4445 , piece_id=LSAB556ASJWW2RIFBJQFNDEBFE2Q66GV4S74ADVEVUJGEPXSN74A, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.175443057 (559h8m50.046526102s) [noise=true , address=77.90.31.10:4517 , piece_id=KG5UVS6GY5A2J7R66MLRVNPW5XORQQWZO2M5FA5FMFAAIQTQ76TQ, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.176789310 (559h8m50.047872345s) [noise=true , address=145.79.227.42:4122 , piece_id=ZF7TATOOVKOVA6MFUWEFEHR2JDC6SVHCAA4TIAEQH7RUT5OKGZAA, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.178271662 (559h8m50.049354717s) [noise=true , address=145.79.227.47:4267 , piece_id=ASZ4QGGW5E6A2GCGOGIQZYHEIDTBP3W467FFR6OAUO6Z7XDPTBCA, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.178602352 (559h8m50.049685387s) [noise=true , address=45.91.116.6:4973 , piece_id=BSHPBQ2KDFJ2LHQTNQXZ5GTYUXPDMPIH2Z7DU5PDMVUQWWZLCWHA, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.179902097 (559h8m50.050985142s) [noise=true , address=45.91.116.7:4010 , piece_id=TCZVIAFRE2S4UIBCEE56LM3K5HVUBWGCRD2QOX6HQNQ2OSWLW7LQ, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.180446162 (559h8m50.051529197s) [noise=true , address=145.79.227.38:4978 , piece_id=DZLINN5SAEKSHBUKWA33QHBJEULTXI7DXUWZFEM77K6UHADRAGNQ, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:51.884284881 (559h8m51.755367926s) [upload=490202]: Next returned nil segment Dec 28 22:37:51.884347030 (559h8m51.755430085s) [upload=490202]: Waiting for error group managing segments... Dec 28 22:37:52.166533359 (559h8m52.037616394s) [noise=true , address=145.79.227.16:4912 , piece_id=J5CR6HNRR62DKVGL4O2P45CFC4LSKMPARVRZ4D7IU3RRYDCKMU4A, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.171377762 (559h8m52.042460808s) [noise=true , address=145.79.227.21:4473 , piece_id=UL5ANBOVC2E3VFYNUT3H2BNSX45FWGWF73HT3PTQ6FOSJC2GUOAA, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.174870432 (559h8m52.045953467s) [noise=true , address=145.79.227.4:4187 , piece_id=VDGZC2AVKT3HCMQSFLN4P3EGZJRSUG4VRFFC7UQXXCBHRVJPICBQ, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.179490749 (559h8m52.050573784s) [noise=true , address=145.79.227.42:4122 , piece_id=ZF7TATOOVKOVA6MFUWEFEHR2JDC6SVHCAA4TIAEQH7RUT5OKGZAA, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.182257707 (559h8m52.053340742s) [noise=true , address=145.79.227.2:4114 , piece_id=6TJZJMIEIIFKPLU6ANH23JICEKVJ4P3KBPQSSJYUUUWDDDPQWQLQ, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.184092781 (559h8m52.055175816s) [noise=true , address=145.79.227.13:4802 , piece_id=7SY56FHEGZ4TZN2SIUKP4I5PG2EMJF4AOS52A2O6ELXRYUTPTLLA, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.188631442 (559h8m52.059714487s) [noise=true , address=145.79.227.52:4445 , piece_id=LSAB556ASJWW2RIFBJQFNDEBFE2Q66GV4S74ADVEVUJGEPXSN74A, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.190834096 (559h8m52.061917131s) [noise=true , address=145.79.227.50:4374 , piece_id=KATXKHW5L3BXASF7MNQU2LOTHZ2GPDBOFL4VGMY4BYNJTPW3LYOQ, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.195035023 (559h8m52.066118079s) [noise=true , address=145.79.227.47:4267 , piece_id=ASZ4QGGW5E6A2GCGOGIQZYHEIDTBP3W467FFR6OAUO6Z7XDPTBCA, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.222064916 (559h8m52.093147952s) [noise=true , address=145.79.227.38:4978 , piece_id=DZLINN5SAEKSHBUKWA33QHBJEULTXI7DXUWZFEM77K6UHADRAGNQ, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.370576240 (559h8m52.241659275s) [noise=true , address=77.90.31.10:4517 , piece_id=KG5UVS6GY5A2J7R66MLRVNPW5XORQQWZO2M5FA5FMFAAIQTQ76TQ, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.412861888 (559h8m52.283944933s) [noise=true , address=45.91.116.6:4973 , piece_id=BSHPBQ2KDFJ2LHQTNQXZ5GTYUXPDMPIH2Z7DU5PDMVUQWWZLCWHA, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.419383675 (559h8m52.29046671s) [noise=true , address=45.91.116.12:4226 , piece_id=Z22UGEYHKTKDY5Q4QBWBFUBFEPOWMKWPIZMM323RILMFYBHM5ECA, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.419405267 (559h8m52.290488312s) [seg_pos={0 0}, upload=490202]: Segment reached optimal threshold of 13 pieces. Dec 28 22:37:52.419679719 (559h8m52.290762755s) [noise=true , address=45.91.116.7:4010 , piece_id=TCZVIAFRE2S4UIBCEE56LM3K5HVUBWGCRD2QOX6HQNQ2OSWLW7LQ, seg_pos={0 0}, upload=490202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 28 22:37:52.419687264 (559h8m52.290770309s) [noise=true , address=45.91.116.7:4010 , piece_id=TCZVIAFRE2S4UIBCEE56LM3K5HVUBWGCRD2QOX6HQNQ2OSWLW7LQ, seg_pos={0 0}, upload=490202]: Long tail cancellation due to optimal threshold Dec 28 22:37:52.419742578 (559h8m52.290825623s) [noise=true , address=77.90.31.21:4880 , piece_id=LSKLRPZVVODLA2W4JX2NCOGEVAQKHFNFSNBRJZE3QN2J5MVVNIBA, seg_pos={0 0}, upload=490202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 28 22:37:52.419763569 (559h8m52.290846604s) [noise=true , address=77.90.31.21:4880 , piece_id=LSKLRPZVVODLA2W4JX2NCOGEVAQKHFNFSNBRJZE3QN2J5MVVNIBA, seg_pos={0 0}, upload=490202]: Long tail cancellation due to optimal threshold Dec 28 22:37:52.419998967 (559h8m52.291082002s) [noise=true , address=145.79.227.5:4222 , piece_id=L5GHMXVRGRNG7MRAXDQ5JBAKGWIH32SA5QQCU6UHOQ5O3POTXY4A, seg_pos={0 0}, upload=490202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:42510->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42510->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42510->145.79.227.5:4222: use of closed network connection Dec 28 22:37:52.420008886 (559h8m52.291091921s) [noise=true , address=145.79.227.5:4222 , piece_id=L5GHMXVRGRNG7MRAXDQ5JBAKGWIH32SA5QQCU6UHOQ5O3POTXY4A, seg_pos={0 0}, upload=490202]: Long tail cancellation due to optimal threshold Dec 28 22:37:52.420087535 (559h8m52.29117058s) [seg_pos={0 0}, upload=490202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:37:52.420366096 (559h8m52.291449131s) [upload=490202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:37:52.420373359 (559h8m52.291456404s) [upload=490202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:37:52.734541670 (559h8m52.605624715s) [upload=490202]: Done uploading segments... Dec 28 22:37:52.734570896 (559h8m52.605653941s) [upload=490202]: Upload finished. err: <nil> Duration: 5.738202444s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X.X...................X...] duration: 15.415514771s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:39:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 15.415544909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X.X...................X..] duration: 33.552671634s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:38:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 33.552699355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................X....................X.....X....X] duration: 30.008815813s trace: https://jaeger.dev.storj.io/trace/5f5562a3131aec10 FAIL (2025-12-28 22:38:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: KG5UVS6GY5A2J7R66MLRVNPW5XORQQWZO2M5FA5FMFAAIQTQ76TQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: BSHPBQ2KDFJ2LHQTNQXZ5GTYUXPDMPIH2Z7DU5PDMVUQWWZLCWHA) context deadline exceeded ======LOGS====== Dec 28 22:37:49.917195004 (559h8m49.788278039s) [upload=490202]: Starting upload Dec 28 22:37:49.917247603 (559h8m49.788330638s) [upload=490202]: Done starting upload Dec 28 22:37:49.917366641 (559h8m49.788449686s) [upload=490202]: Uploading segments... Dec 28 22:37:49.917459407 (559h8m49.788542442s) [upload=490202]: Got next segment. Inline: false Dec 28 22:37:49.917492500 (559h8m49.788575545s) [upload=490202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:37:49.917497270 (559h8m49.788580315s) [upload=490202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:37:50.166914830 (559h8m50.037997865s) [seg_pos={0 0}, upload=490202]: Begin upload segment... Dec 28 22:37:50.166965275 (559h8m50.03804831s) [seg_pos={0 0}, upload=490202]: Done begin upload segment. Dec 28 22:37:50.167009459 (559h8m50.038092494s) [noise=true , address=145.79.227.50:4374 , piece_id=KATXKHW5L3BXASF7MNQU2LOTHZ2GPDBOFL4VGMY4BYNJTPW3LYOQ, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.167066328 (559h8m50.038149393s) [noise=true , address=145.79.227.16:4912 , piece_id=J5CR6HNRR62DKVGL4O2P45CFC4LSKMPARVRZ4D7IU3RRYDCKMU4A, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.167686259 (559h8m50.038769294s) [noise=true , address=145.79.227.21:4473 , piece_id=UL5ANBOVC2E3VFYNUT3H2BNSX45FWGWF73HT3PTQ6FOSJC2GUOAA, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.168137247 (559h8m50.039220282s) [noise=true , address=145.79.227.2:4114 , piece_id=6TJZJMIEIIFKPLU6ANH23JICEKVJ4P3KBPQSSJYUUUWDDDPQWQLQ, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.168565432 (559h8m50.039648477s) [noise=true , address=77.90.31.21:4880 , piece_id=LSKLRPZVVODLA2W4JX2NCOGEVAQKHFNFSNBRJZE3QN2J5MVVNIBA, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.169995636 (559h8m50.041078681s) [noise=true , address=145.79.227.5:4222 , piece_id=L5GHMXVRGRNG7MRAXDQ5JBAKGWIH32SA5QQCU6UHOQ5O3POTXY4A, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.170943240 (559h8m50.042026275s) [noise=true , address=145.79.227.13:4802 , piece_id=7SY56FHEGZ4TZN2SIUKP4I5PG2EMJF4AOS52A2O6ELXRYUTPTLLA, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.171223303 (559h8m50.042306348s) [noise=true , address=145.79.227.4:4187 , piece_id=VDGZC2AVKT3HCMQSFLN4P3EGZJRSUG4VRFFC7UQXXCBHRVJPICBQ, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.172627797 (559h8m50.043710832s) [noise=true , address=45.91.116.12:4226 , piece_id=Z22UGEYHKTKDY5Q4QBWBFUBFEPOWMKWPIZMM323RILMFYBHM5ECA, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.173903606 (559h8m50.044986641s) [noise=true , address=145.79.227.52:4445 , piece_id=LSAB556ASJWW2RIFBJQFNDEBFE2Q66GV4S74ADVEVUJGEPXSN74A, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.175443057 (559h8m50.046526102s) [noise=true , address=77.90.31.10:4517 , piece_id=KG5UVS6GY5A2J7R66MLRVNPW5XORQQWZO2M5FA5FMFAAIQTQ76TQ, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.176789310 (559h8m50.047872345s) [noise=true , address=145.79.227.42:4122 , piece_id=ZF7TATOOVKOVA6MFUWEFEHR2JDC6SVHCAA4TIAEQH7RUT5OKGZAA, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.178271662 (559h8m50.049354717s) [noise=true , address=145.79.227.47:4267 , piece_id=ASZ4QGGW5E6A2GCGOGIQZYHEIDTBP3W467FFR6OAUO6Z7XDPTBCA, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.178602352 (559h8m50.049685387s) [noise=true , address=45.91.116.6:4973 , piece_id=BSHPBQ2KDFJ2LHQTNQXZ5GTYUXPDMPIH2Z7DU5PDMVUQWWZLCWHA, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.179902097 (559h8m50.050985142s) [noise=true , address=45.91.116.7:4010 , piece_id=TCZVIAFRE2S4UIBCEE56LM3K5HVUBWGCRD2QOX6HQNQ2OSWLW7LQ, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:50.180446162 (559h8m50.051529197s) [noise=true , address=145.79.227.38:4978 , piece_id=DZLINN5SAEKSHBUKWA33QHBJEULTXI7DXUWZFEM77K6UHADRAGNQ, seg_pos={0 0}, upload=490202]: Uploading piece... Dec 28 22:37:51.884284881 (559h8m51.755367926s) [upload=490202]: Next returned nil segment Dec 28 22:37:51.884347030 (559h8m51.755430085s) [upload=490202]: Waiting for error group managing segments... Dec 28 22:37:52.166533359 (559h8m52.037616394s) [noise=true , address=145.79.227.16:4912 , piece_id=J5CR6HNRR62DKVGL4O2P45CFC4LSKMPARVRZ4D7IU3RRYDCKMU4A, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.171377762 (559h8m52.042460808s) [noise=true , address=145.79.227.21:4473 , piece_id=UL5ANBOVC2E3VFYNUT3H2BNSX45FWGWF73HT3PTQ6FOSJC2GUOAA, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.174870432 (559h8m52.045953467s) [noise=true , address=145.79.227.4:4187 , piece_id=VDGZC2AVKT3HCMQSFLN4P3EGZJRSUG4VRFFC7UQXXCBHRVJPICBQ, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.179490749 (559h8m52.050573784s) [noise=true , address=145.79.227.42:4122 , piece_id=ZF7TATOOVKOVA6MFUWEFEHR2JDC6SVHCAA4TIAEQH7RUT5OKGZAA, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.182257707 (559h8m52.053340742s) [noise=true , address=145.79.227.2:4114 , piece_id=6TJZJMIEIIFKPLU6ANH23JICEKVJ4P3KBPQSSJYUUUWDDDPQWQLQ, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.184092781 (559h8m52.055175816s) [noise=true , address=145.79.227.13:4802 , piece_id=7SY56FHEGZ4TZN2SIUKP4I5PG2EMJF4AOS52A2O6ELXRYUTPTLLA, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.188631442 (559h8m52.059714487s) [noise=true , address=145.79.227.52:4445 , piece_id=LSAB556ASJWW2RIFBJQFNDEBFE2Q66GV4S74ADVEVUJGEPXSN74A, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.190834096 (559h8m52.061917131s) [noise=true , address=145.79.227.50:4374 , piece_id=KATXKHW5L3BXASF7MNQU2LOTHZ2GPDBOFL4VGMY4BYNJTPW3LYOQ, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.195035023 (559h8m52.066118079s) [noise=true , address=145.79.227.47:4267 , piece_id=ASZ4QGGW5E6A2GCGOGIQZYHEIDTBP3W467FFR6OAUO6Z7XDPTBCA, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.222064916 (559h8m52.093147952s) [noise=true , address=145.79.227.38:4978 , piece_id=DZLINN5SAEKSHBUKWA33QHBJEULTXI7DXUWZFEM77K6UHADRAGNQ, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.370576240 (559h8m52.241659275s) [noise=true , address=77.90.31.10:4517 , piece_id=KG5UVS6GY5A2J7R66MLRVNPW5XORQQWZO2M5FA5FMFAAIQTQ76TQ, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.412861888 (559h8m52.283944933s) [noise=true , address=45.91.116.6:4973 , piece_id=BSHPBQ2KDFJ2LHQTNQXZ5GTYUXPDMPIH2Z7DU5PDMVUQWWZLCWHA, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.419383675 (559h8m52.29046671s) [noise=true , address=45.91.116.12:4226 , piece_id=Z22UGEYHKTKDY5Q4QBWBFUBFEPOWMKWPIZMM323RILMFYBHM5ECA, seg_pos={0 0}, upload=490202]: Done uploading piece. err: <nil> Dec 28 22:37:52.419405267 (559h8m52.290488312s) [seg_pos={0 0}, upload=490202]: Segment reached optimal threshold of 13 pieces. Dec 28 22:37:52.419679719 (559h8m52.290762755s) [noise=true , address=45.91.116.7:4010 , piece_id=TCZVIAFRE2S4UIBCEE56LM3K5HVUBWGCRD2QOX6HQNQ2OSWLW7LQ, seg_pos={0 0}, upload=490202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 28 22:37:52.419687264 (559h8m52.290770309s) [noise=true , address=45.91.116.7:4010 , piece_id=TCZVIAFRE2S4UIBCEE56LM3K5HVUBWGCRD2QOX6HQNQ2OSWLW7LQ, seg_pos={0 0}, upload=490202]: Long tail cancellation due to optimal threshold Dec 28 22:37:52.419742578 (559h8m52.290825623s) [noise=true , address=77.90.31.21:4880 , piece_id=LSKLRPZVVODLA2W4JX2NCOGEVAQKHFNFSNBRJZE3QN2J5MVVNIBA, seg_pos={0 0}, upload=490202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 28 22:37:52.419763569 (559h8m52.290846604s) [noise=true , address=77.90.31.21:4880 , piece_id=LSKLRPZVVODLA2W4JX2NCOGEVAQKHFNFSNBRJZE3QN2J5MVVNIBA, seg_pos={0 0}, upload=490202]: Long tail cancellation due to optimal threshold Dec 28 22:37:52.419998967 (559h8m52.291082002s) [noise=true , address=145.79.227.5:4222 , piece_id=L5GHMXVRGRNG7MRAXDQ5JBAKGWIH32SA5QQCU6UHOQ5O3POTXY4A, seg_pos={0 0}, upload=490202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:42510->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42510->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42510->145.79.227.5:4222: use of closed network connection Dec 28 22:37:52.420008886 (559h8m52.291091921s) [noise=true , address=145.79.227.5:4222 , piece_id=L5GHMXVRGRNG7MRAXDQ5JBAKGWIH32SA5QQCU6UHOQ5O3POTXY4A, seg_pos={0 0}, upload=490202]: Long tail cancellation due to optimal threshold Dec 28 22:37:52.420087535 (559h8m52.29117058s) [seg_pos={0 0}, upload=490202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:37:52.420366096 (559h8m52.291449131s) [upload=490202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:37:52.420373359 (559h8m52.291456404s) [upload=490202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:37:52.734541670 (559h8m52.605624715s) [upload=490202]: Done uploading segments... Dec 28 22:37:52.734570896 (559h8m52.605653941s) [upload=490202]: Upload finished. err: <nil> Duration: 30.008837173s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X.X...................X.] duration: 25.191049052s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:37:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 25.191075643s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.X...................X] duration: 1m0.002993192s trace: https://jaeger.dev.storj.io/trace/5789d7e0941027a4 FAIL (2025-12-28 22:37:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== pload=490181]: Uploading piece... Dec 28 22:36:15.823907409 (559h7m15.694990444s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.824829695 (559h7m15.69591274s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.825470294 (559h7m15.696553329s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826060939 (559h7m15.697143974s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.826172311 (559h7m15.697255346s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827056584 (559h7m15.698139619s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.827902886 (559h7m15.698985922s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828261168 (559h7m15.699344203s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828322144 (559h7m15.699405179s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.828847664 (559h7m15.699930699s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829263376 (559h7m15.700346421s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.829711640 (559h7m15.700794685s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.830191734 (559h7m15.701274769s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831228277 (559h7m15.702311322s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.831732216 (559h7m15.702815261s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.832150453 (559h7m15.703233488s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:15.835255014 (559h7m15.706338059s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Uploading piece... Dec 28 22:36:17.279694919 (559h7m17.150777954s) [upload=490181]: Next returned nil segment Dec 28 22:36:17.279728523 (559h7m17.150811558s) [upload=490181]: Waiting for error group managing segments... Dec 28 22:36:17.724373871 (559h7m17.595456916s) [noise=true , address=145.79.227.15:4900 , piece_id=RHBXLGDA2YJ5DXO4L475HAFMIJ6BGZJTFDVO6U4GVYB643K7BMZQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.733953008 (559h7m17.605036043s) [noise=true , address=145.79.227.21:4479 , piece_id=KL5MHJGYJOODJA6FN325AGCPQHGEZ743XBPW3LDQQLE6INVBDKCQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.744234914 (559h7m17.615317959s) [noise=true , address=145.79.227.16:4925 , piece_id=MXTIXCSIWRDJBYLT2VYWYJQMXLUG7UUXLA5BQI2CQ64XU54HJ3AQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.908880596 (559h7m17.779963632s) [noise=true , address=77.90.31.14:4687 , piece_id=5QDY6EP4U7LWVIEI5F2KTWHL36QNVQZVWLFRMRYUQ3R7ZP2A6P6A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.928925500 (559h7m17.800008545s) [noise=true , address=77.90.31.21:4910 , piece_id=RNPEZ6WJ7NAH4ZJ3CC2IF6PBOSOJ2DWSLFWASUU3KBTQ5M7IMIOQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.935807983 (559h7m17.806891018s) [noise=true , address=77.90.31.10:4524 , piece_id=AURPAKBVZ2LZLD5FZAX34XTAPVDPVULTMJRJG65VD2ZVUUO7W53A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:17.983414633 (559h7m17.854497668s) [noise=true , address=77.90.31.18:4840 , piece_id=COKXERNAXSUNT3CVIW7LMQYMRGCYY2IZUC3OSBJXR7RAJLYLLRWA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.026382169 (559h7m17.897465204s) [noise=true , address=77.90.31.16:4739 , piece_id=MZSJRAOQVP6PL6CFYSYLNPU4CPQVMWCJDXXLQSO2OCFFSSIN655Q, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.036528587 (559h7m17.907611633s) [noise=true , address=77.90.31.11:4562 , piece_id=WBBFUJL7GJNV4AAWJKS3WY2ONR7CL63BV2BROXAWBH34WR3CYDLQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.037727420 (559h7m17.908810455s) [noise=true , address=77.90.31.13:4642 , piece_id=FSSGQCZC7MKGG6YENCTC7FWXJGXY5DRIANWWE56KEMWLXTG72OMA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.623298990 (559h7m18.494382025s) [noise=true , address=145.79.227.34:4939 , piece_id=K2SCEKFM4A6VSRUWT54RYHIS6V4YNCVPW2RNQ7RVGMMQYGWITJ5A, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.637950695 (559h7m18.50903374s) [noise=true , address=145.79.227.49:4350 , piece_id=2G4ADOF5CEKAAIFLMHRL4NISVC4SD3DAZVFYIUUXW3KIBNTTFCTQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.647934494 (559h7m18.519017529s) [noise=true , address=145.79.227.34:4932 , piece_id=L6WWVHKTXJA3JMUEDSGAJS64C53YPCBRBU4O27BBBHGAV4X3DIIA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690591600 (559h7m18.561674645s) [noise=true , address=145.79.227.44:4221 , piece_id=A6KHSCZCTM5T5SQQBXXXVB4BIQTSRRYAU3OTTGIWUQFS4DAW5EVQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: <nil> Dec 28 22:36:18.690617579 (559h7m18.561700624s) [seg_pos={0 0}, upload=490181]: Segment reached optimal threshold of 14 pieces. Dec 28 22:36:18.690941035 (559h7m18.56202407s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.690960302 (559h7m18.562043337s) [noise=true , address=145.79.227.19:4110 , piece_id=VPVJAYRYVW2SEETXGIVVS6NCWMZCPJHOBJYQK75C6SJLV7ZAXFXQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691032850 (559h7m18.562115895s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691044201 (559h7m18.562127256s) [noise=true , address=77.90.31.15:4726 , piece_id=F3OGLFNOOQZQCCB4SD2OTU4LGA5J3GZI2T5VSK5TWZGPVKFGXGKQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691183346 (559h7m18.562266381s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691190309 (559h7m18.562273344s) [noise=true , address=145.79.227.33:4901 , piece_id=Y5KGW4XL7RQYYDBCT7K6JT3PSU46AVHK2J5YEVKXIE4BUDH4NBHA, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691352448 (559h7m18.562435484s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ): protocol: expected piece hash; context canceled; EOF Dec 28 22:36:18.691360504 (559h7m18.562443549s) [noise=true , address=145.79.227.43:4175 , piece_id=CR2PVFGZENZDKOSTQGXSDRDXHPTZ47WLVHLOUZSCHEEZRHEIS5NQ, seg_pos={0 0}, upload=490181]: Long tail cancellation due to optimal threshold Dec 28 22:36:18.691382225 (559h7m18.56246526s) [seg_pos={0 0}, upload=490181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:36:18.691510559 (559h7m18.562593604s) [upload=490181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:36:18.691519567 (559h7m18.562602612s) [upload=490181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:36:18.994571953 (559h7m18.865654988s) [upload=490181]: Done uploading segments... Dec 28 22:36:18.994593514 (559h7m18.865676559s) [upload=490181]: Upload finished. err: <nil> Duration: 1m0.003014002s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X....................X.....X....] duration: 5.996076406s trace: https://jaeger.dev.storj.io/trace/5e4656c542eca88e FAIL (2025-12-28 22:36:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 4LKIIYWMQ4RTMQVFD3URYEZ4EFLGY6WCRYOMW6AUV6OIYGTJZLEQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: 57FEQYPP7I33MP3HRYFV22BOJ7MNM5TFDN3GRRDIQJ7MUFZR3MJQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: FOWBW36IZCJ3IJYITKT4VKQBR6SVPXY4SOTV3WC5NTHZL74Q4I3A) context deadline exceeded ======LOGS====== Dec 28 22:33:29.729946652 (559h4m29.601029697s) [upload=490140]: Starting upload Dec 28 22:33:29.730011725 (559h4m29.60109476s) [upload=490140]: Done starting upload Dec 28 22:33:29.730059898 (559h4m29.601142933s) [upload=490140]: Uploading segments... Dec 28 22:33:29.730191068 (559h4m29.601274103s) [upload=490140]: Got next segment. Inline: false Dec 28 22:33:29.730225061 (559h4m29.601308096s) [upload=490140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:33:29.730229050 (559h4m29.601312085s) [upload=490140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:33:29.982960409 (559h4m29.854043444s) [seg_pos={0 0}, upload=490140]: Begin upload segment... Dec 28 22:33:29.983015174 (559h4m29.854098209s) [seg_pos={0 0}, upload=490140]: Done begin upload segment. Dec 28 22:33:29.983073906 (559h4m29.854156941s) [noise=true , address=145.79.227.51:4409 , piece_id=ZWRDHUVEM6GHIJHV3RUJO5XJJCZ3ZQTOU5HAK2URT2HK5I5ZXZKA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.983088052 (559h4m29.854171097s) [noise=true , address=145.79.227.26:4654 , piece_id=YTXOCCTG3AVOUZDFPOXEKBZXIOZPCQJ4VSSG7HHQWVT52SZCCGJA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.983688816 (559h4m29.854771851s) [noise=true , address=145.79.227.38:4979 , piece_id=TFPL7BNNFWOGW6ZB4AE5437CZ2U7TCD4SSQHTMJ4PU27B2RDFS3Q, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.984142400 (559h4m29.855225445s) [noise=true , address=145.79.227.39:4015 , piece_id=57FEQYPP7I33MP3HRYFV22BOJ7MNM5TFDN3GRRDIQJ7MUFZR3MJQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.984532923 (559h4m29.855615958s) [noise=true , address=45.91.116.8:4045 , piece_id=NFOF2OA7JLTOVAHY4BC54OKNACCNP36U6DPDQWNYRNX3NK7JZSZA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.984876387 (559h4m29.855959422s) [noise=true , address=45.91.116.7:4012 , piece_id=FOWBW36IZCJ3IJYITKT4VKQBR6SVPXY4SOTV3WC5NTHZL74Q4I3A, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.985415613 (559h4m29.856498659s) [noise=true , address=145.79.227.15:4875 , piece_id=JWMDBQGJY2LTQGCBYOXMFF3FM7SIYR6W6LCPETCRWIOOXNXZYWPA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.985952105 (559h4m29.85703515s) [noise=true , address=45.91.116.12:4225 , piece_id=4LKIIYWMQ4RTMQVFD3URYEZ4EFLGY6WCRYOMW6AUV6OIYGTJZLEQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.986963731 (559h4m29.858046766s) [noise=true , address=145.79.227.19:4078 , piece_id=LQWLLQ2OIXPJHB6WTNAWPNLBIDTZEGSCHMX5DSVCCXRB4YJMJPXQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.987187446 (559h4m29.858270482s) [noise=true , address=45.91.116.6:4974 , piece_id=6UFJLFBEKPNQ7U47GD5XRP4L47HO3DJPMAPRLFXTJXK47KIBTBAQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.988386640 (559h4m29.859469676s) [noise=true , address=145.79.227.50:4374 , piece_id=DQCLZ5UZUKXHYK5MNWC7GBCR6RXAYOUFWGKFSSTBYQSVAS7MR47A, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.988736587 (559h4m29.859819632s) [noise=true , address=45.91.116.9:4261 , piece_id=YWFQCOVU6G2MVFYNPUL37MXC5TEOOHY3PZH3RM5RMTBVQRLIABDQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.989595411 (559h4m29.860678447s) [noise=true , address=145.79.227.48:4301 , piece_id=2FNMGYKOCOSK7OCNSKSLPX6GIRT6PFIL7VI6FTWR5WEALWYRY4RQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.990731414 (559h4m29.861814459s) [noise=true , address=145.79.227.37:4566 , piece_id=UVVMZOAZWTONSFF5ODSBJ3UZB2NMJMCI7VK76A24HIOFHQ5X3WQA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.993898394 (559h4m29.86498143s) [noise=true , address=77.90.31.21:4878 , piece_id=TBJZYLI7R2CE7JY7GPDTPXDITWAZIVC2TLJ6XESE6OZ343OZKWXA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:30.000210902 (559h4m29.871293957s) [noise=true , address=145.79.227.46:4601 , piece_id=P6H3HYQHUEZZZHTSFCHV6CSJ2C75XEBRIT3EZDTDCCGJYLI5RRBA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:31.595685625 (559h4m31.466768671s) [upload=490140]: Next returned nil segment Dec 28 22:33:31.595703990 (559h4m31.466787035s) [upload=490140]: Waiting for error group managing segments... Dec 28 22:33:31.976431013 (559h4m31.847514058s) [noise=true , address=145.79.227.26:4654 , piece_id=YTXOCCTG3AVOUZDFPOXEKBZXIOZPCQJ4VSSG7HHQWVT52SZCCGJA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:31.992221107 (559h4m31.863304152s) [noise=true , address=145.79.227.51:4409 , piece_id=ZWRDHUVEM6GHIJHV3RUJO5XJJCZ3ZQTOU5HAK2URT2HK5I5ZXZKA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.000376273 (559h4m31.871459318s) [noise=true , address=145.79.227.19:4078 , piece_id=LQWLLQ2OIXPJHB6WTNAWPNLBIDTZEGSCHMX5DSVCCXRB4YJMJPXQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.002835345 (559h4m31.8739184s) [noise=true , address=145.79.227.37:4566 , piece_id=UVVMZOAZWTONSFF5ODSBJ3UZB2NMJMCI7VK76A24HIOFHQ5X3WQA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.004171389 (559h4m31.875254424s) [noise=true , address=145.79.227.39:4015 , piece_id=57FEQYPP7I33MP3HRYFV22BOJ7MNM5TFDN3GRRDIQJ7MUFZR3MJQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.007413442 (559h4m31.878496487s) [noise=true , address=145.79.227.50:4374 , piece_id=DQCLZ5UZUKXHYK5MNWC7GBCR6RXAYOUFWGKFSSTBYQSVAS7MR47A, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.014865720 (559h4m31.885948755s) [noise=true , address=145.79.227.38:4979 , piece_id=TFPL7BNNFWOGW6ZB4AE5437CZ2U7TCD4SSQHTMJ4PU27B2RDFS3Q, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.017462344 (559h4m31.888545379s) [noise=true , address=145.79.227.46:4601 , piece_id=P6H3HYQHUEZZZHTSFCHV6CSJ2C75XEBRIT3EZDTDCCGJYLI5RRBA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.018098896 (559h4m31.889181941s) [noise=true , address=145.79.227.15:4875 , piece_id=JWMDBQGJY2LTQGCBYOXMFF3FM7SIYR6W6LCPETCRWIOOXNXZYWPA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.020611821 (559h4m31.891694856s) [noise=true , address=145.79.227.48:4301 , piece_id=2FNMGYKOCOSK7OCNSKSLPX6GIRT6PFIL7VI6FTWR5WEALWYRY4RQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.224145070 (559h4m32.095228106s) [noise=true , address=45.91.116.12:4225 , piece_id=4LKIIYWMQ4RTMQVFD3URYEZ4EFLGY6WCRYOMW6AUV6OIYGTJZLEQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.259872001 (559h4m32.130955046s) [noise=true , address=77.90.31.21:4878 , piece_id=TBJZYLI7R2CE7JY7GPDTPXDITWAZIVC2TLJ6XESE6OZ343OZKWXA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.272662344 (559h4m32.143745389s) [noise=true , address=45.91.116.9:4261 , piece_id=YWFQCOVU6G2MVFYNPUL37MXC5TEOOHY3PZH3RM5RMTBVQRLIABDQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.272684415 (559h4m32.14376745s) [seg_pos={0 0}, upload=490140]: Segment reached optimal threshold of 13 pieces. Dec 28 22:33:32.272850472 (559h4m32.143933507s) [noise=true , address=45.91.116.6:4974 , piece_id=6UFJLFBEKPNQ7U47GD5XRP4L47HO3DJPMAPRLFXTJXK47KIBTBAQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:44822->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:44822->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44822->45.91.116.6:4974: use of closed network connection Dec 28 22:33:32.272856653 (559h4m32.143939688s) [noise=true , address=45.91.116.6:4974 , piece_id=6UFJLFBEKPNQ7U47GD5XRP4L47HO3DJPMAPRLFXTJXK47KIBTBAQ, seg_pos={0 0}, upload=490140]: Long tail cancellation due to optimal threshold Dec 28 22:33:32.273054170 (559h4m32.144137205s) [noise=true , address=45.91.116.7:4012 , piece_id=FOWBW36IZCJ3IJYITKT4VKQBR6SVPXY4SOTV3WC5NTHZL74Q4I3A, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.273116308 (559h4m32.144199353s) [noise=true , address=45.91.116.8:4045 , piece_id=NFOF2OA7JLTOVAHY4BC54OKNACCNP36U6DPDQWNYRNX3NK7JZSZA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 22:33:32.273123332 (559h4m32.144206377s) [noise=true , address=45.91.116.8:4045 , piece_id=NFOF2OA7JLTOVAHY4BC54OKNACCNP36U6DPDQWNYRNX3NK7JZSZA, seg_pos={0 0}, upload=490140]: Long tail cancellation due to optimal threshold Dec 28 22:33:32.273145975 (559h4m32.14422901s) [seg_pos={0 0}, upload=490140]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 28 22:33:32.273254051 (559h4m32.144337086s) [upload=490140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:33:32.273259291 (559h4m32.144342326s) [upload=490140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:33:32.568117145 (559h4m32.43920019s) [upload=490140]: Done uploading segments... Dec 28 22:33:32.568136302 (559h4m32.439219347s) [upload=490140]: Upload finished. err: <nil> Duration: 5.996102006s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X....................X.....X...] duration: 6.085690042s trace: https://jaeger.dev.storj.io/trace/5e4656c542eca88e FAIL (2025-12-28 22:36:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 4LKIIYWMQ4RTMQVFD3URYEZ4EFLGY6WCRYOMW6AUV6OIYGTJZLEQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: 57FEQYPP7I33MP3HRYFV22BOJ7MNM5TFDN3GRRDIQJ7MUFZR3MJQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: FOWBW36IZCJ3IJYITKT4VKQBR6SVPXY4SOTV3WC5NTHZL74Q4I3A) context deadline exceeded ======LOGS====== Dec 28 22:33:29.729946652 (559h4m29.601029697s) [upload=490140]: Starting upload Dec 28 22:33:29.730011725 (559h4m29.60109476s) [upload=490140]: Done starting upload Dec 28 22:33:29.730059898 (559h4m29.601142933s) [upload=490140]: Uploading segments... Dec 28 22:33:29.730191068 (559h4m29.601274103s) [upload=490140]: Got next segment. Inline: false Dec 28 22:33:29.730225061 (559h4m29.601308096s) [upload=490140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:33:29.730229050 (559h4m29.601312085s) [upload=490140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:33:29.982960409 (559h4m29.854043444s) [seg_pos={0 0}, upload=490140]: Begin upload segment... Dec 28 22:33:29.983015174 (559h4m29.854098209s) [seg_pos={0 0}, upload=490140]: Done begin upload segment. Dec 28 22:33:29.983073906 (559h4m29.854156941s) [noise=true , address=145.79.227.51:4409 , piece_id=ZWRDHUVEM6GHIJHV3RUJO5XJJCZ3ZQTOU5HAK2URT2HK5I5ZXZKA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.983088052 (559h4m29.854171097s) [noise=true , address=145.79.227.26:4654 , piece_id=YTXOCCTG3AVOUZDFPOXEKBZXIOZPCQJ4VSSG7HHQWVT52SZCCGJA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.983688816 (559h4m29.854771851s) [noise=true , address=145.79.227.38:4979 , piece_id=TFPL7BNNFWOGW6ZB4AE5437CZ2U7TCD4SSQHTMJ4PU27B2RDFS3Q, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.984142400 (559h4m29.855225445s) [noise=true , address=145.79.227.39:4015 , piece_id=57FEQYPP7I33MP3HRYFV22BOJ7MNM5TFDN3GRRDIQJ7MUFZR3MJQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.984532923 (559h4m29.855615958s) [noise=true , address=45.91.116.8:4045 , piece_id=NFOF2OA7JLTOVAHY4BC54OKNACCNP36U6DPDQWNYRNX3NK7JZSZA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.984876387 (559h4m29.855959422s) [noise=true , address=45.91.116.7:4012 , piece_id=FOWBW36IZCJ3IJYITKT4VKQBR6SVPXY4SOTV3WC5NTHZL74Q4I3A, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.985415613 (559h4m29.856498659s) [noise=true , address=145.79.227.15:4875 , piece_id=JWMDBQGJY2LTQGCBYOXMFF3FM7SIYR6W6LCPETCRWIOOXNXZYWPA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.985952105 (559h4m29.85703515s) [noise=true , address=45.91.116.12:4225 , piece_id=4LKIIYWMQ4RTMQVFD3URYEZ4EFLGY6WCRYOMW6AUV6OIYGTJZLEQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.986963731 (559h4m29.858046766s) [noise=true , address=145.79.227.19:4078 , piece_id=LQWLLQ2OIXPJHB6WTNAWPNLBIDTZEGSCHMX5DSVCCXRB4YJMJPXQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.987187446 (559h4m29.858270482s) [noise=true , address=45.91.116.6:4974 , piece_id=6UFJLFBEKPNQ7U47GD5XRP4L47HO3DJPMAPRLFXTJXK47KIBTBAQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.988386640 (559h4m29.859469676s) [noise=true , address=145.79.227.50:4374 , piece_id=DQCLZ5UZUKXHYK5MNWC7GBCR6RXAYOUFWGKFSSTBYQSVAS7MR47A, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.988736587 (559h4m29.859819632s) [noise=true , address=45.91.116.9:4261 , piece_id=YWFQCOVU6G2MVFYNPUL37MXC5TEOOHY3PZH3RM5RMTBVQRLIABDQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.989595411 (559h4m29.860678447s) [noise=true , address=145.79.227.48:4301 , piece_id=2FNMGYKOCOSK7OCNSKSLPX6GIRT6PFIL7VI6FTWR5WEALWYRY4RQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.990731414 (559h4m29.861814459s) [noise=true , address=145.79.227.37:4566 , piece_id=UVVMZOAZWTONSFF5ODSBJ3UZB2NMJMCI7VK76A24HIOFHQ5X3WQA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.993898394 (559h4m29.86498143s) [noise=true , address=77.90.31.21:4878 , piece_id=TBJZYLI7R2CE7JY7GPDTPXDITWAZIVC2TLJ6XESE6OZ343OZKWXA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:30.000210902 (559h4m29.871293957s) [noise=true , address=145.79.227.46:4601 , piece_id=P6H3HYQHUEZZZHTSFCHV6CSJ2C75XEBRIT3EZDTDCCGJYLI5RRBA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:31.595685625 (559h4m31.466768671s) [upload=490140]: Next returned nil segment Dec 28 22:33:31.595703990 (559h4m31.466787035s) [upload=490140]: Waiting for error group managing segments... Dec 28 22:33:31.976431013 (559h4m31.847514058s) [noise=true , address=145.79.227.26:4654 , piece_id=YTXOCCTG3AVOUZDFPOXEKBZXIOZPCQJ4VSSG7HHQWVT52SZCCGJA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:31.992221107 (559h4m31.863304152s) [noise=true , address=145.79.227.51:4409 , piece_id=ZWRDHUVEM6GHIJHV3RUJO5XJJCZ3ZQTOU5HAK2URT2HK5I5ZXZKA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.000376273 (559h4m31.871459318s) [noise=true , address=145.79.227.19:4078 , piece_id=LQWLLQ2OIXPJHB6WTNAWPNLBIDTZEGSCHMX5DSVCCXRB4YJMJPXQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.002835345 (559h4m31.8739184s) [noise=true , address=145.79.227.37:4566 , piece_id=UVVMZOAZWTONSFF5ODSBJ3UZB2NMJMCI7VK76A24HIOFHQ5X3WQA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.004171389 (559h4m31.875254424s) [noise=true , address=145.79.227.39:4015 , piece_id=57FEQYPP7I33MP3HRYFV22BOJ7MNM5TFDN3GRRDIQJ7MUFZR3MJQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.007413442 (559h4m31.878496487s) [noise=true , address=145.79.227.50:4374 , piece_id=DQCLZ5UZUKXHYK5MNWC7GBCR6RXAYOUFWGKFSSTBYQSVAS7MR47A, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.014865720 (559h4m31.885948755s) [noise=true , address=145.79.227.38:4979 , piece_id=TFPL7BNNFWOGW6ZB4AE5437CZ2U7TCD4SSQHTMJ4PU27B2RDFS3Q, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.017462344 (559h4m31.888545379s) [noise=true , address=145.79.227.46:4601 , piece_id=P6H3HYQHUEZZZHTSFCHV6CSJ2C75XEBRIT3EZDTDCCGJYLI5RRBA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.018098896 (559h4m31.889181941s) [noise=true , address=145.79.227.15:4875 , piece_id=JWMDBQGJY2LTQGCBYOXMFF3FM7SIYR6W6LCPETCRWIOOXNXZYWPA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.020611821 (559h4m31.891694856s) [noise=true , address=145.79.227.48:4301 , piece_id=2FNMGYKOCOSK7OCNSKSLPX6GIRT6PFIL7VI6FTWR5WEALWYRY4RQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.224145070 (559h4m32.095228106s) [noise=true , address=45.91.116.12:4225 , piece_id=4LKIIYWMQ4RTMQVFD3URYEZ4EFLGY6WCRYOMW6AUV6OIYGTJZLEQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.259872001 (559h4m32.130955046s) [noise=true , address=77.90.31.21:4878 , piece_id=TBJZYLI7R2CE7JY7GPDTPXDITWAZIVC2TLJ6XESE6OZ343OZKWXA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.272662344 (559h4m32.143745389s) [noise=true , address=45.91.116.9:4261 , piece_id=YWFQCOVU6G2MVFYNPUL37MXC5TEOOHY3PZH3RM5RMTBVQRLIABDQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.272684415 (559h4m32.14376745s) [seg_pos={0 0}, upload=490140]: Segment reached optimal threshold of 13 pieces. Dec 28 22:33:32.272850472 (559h4m32.143933507s) [noise=true , address=45.91.116.6:4974 , piece_id=6UFJLFBEKPNQ7U47GD5XRP4L47HO3DJPMAPRLFXTJXK47KIBTBAQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:44822->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:44822->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44822->45.91.116.6:4974: use of closed network connection Dec 28 22:33:32.272856653 (559h4m32.143939688s) [noise=true , address=45.91.116.6:4974 , piece_id=6UFJLFBEKPNQ7U47GD5XRP4L47HO3DJPMAPRLFXTJXK47KIBTBAQ, seg_pos={0 0}, upload=490140]: Long tail cancellation due to optimal threshold Dec 28 22:33:32.273054170 (559h4m32.144137205s) [noise=true , address=45.91.116.7:4012 , piece_id=FOWBW36IZCJ3IJYITKT4VKQBR6SVPXY4SOTV3WC5NTHZL74Q4I3A, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.273116308 (559h4m32.144199353s) [noise=true , address=45.91.116.8:4045 , piece_id=NFOF2OA7JLTOVAHY4BC54OKNACCNP36U6DPDQWNYRNX3NK7JZSZA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 22:33:32.273123332 (559h4m32.144206377s) [noise=true , address=45.91.116.8:4045 , piece_id=NFOF2OA7JLTOVAHY4BC54OKNACCNP36U6DPDQWNYRNX3NK7JZSZA, seg_pos={0 0}, upload=490140]: Long tail cancellation due to optimal threshold Dec 28 22:33:32.273145975 (559h4m32.14422901s) [seg_pos={0 0}, upload=490140]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 28 22:33:32.273254051 (559h4m32.144337086s) [upload=490140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:33:32.273259291 (559h4m32.144342326s) [upload=490140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:33:32.568117145 (559h4m32.43920019s) [upload=490140]: Done uploading segments... Dec 28 22:33:32.568136302 (559h4m32.439219347s) [upload=490140]: Upload finished. err: <nil> Duration: 6.085718826s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.X...................] duration: 19.513761897s trace: https://jaeger.dev.storj.io/trace/2dde5fb84d6f9aef FAIL (2025-12-28 22:35:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: 7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA) context deadline exceeded; piecestore: (Node ID: 1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, Piece ID: TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 28 22:21:13.466126077 (558h52m13.337209112s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466130635 (558h52m13.33721367s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466584950 (558h52m13.337667985s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466681324 (558h52m13.337764369s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466954685 (558h52m13.33803772s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467283190 (558h52m13.338366235s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467651912 (558h52m13.338734957s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467926144 (558h52m13.33900918s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468224863 (558h52m13.339307908s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468533250 (558h52m13.339616295s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468852889 (558h52m13.339935924s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.474952334 (558h52m13.346035369s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.476142490 (558h52m13.347225525s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.477282110 (558h52m13.348365145s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.478719176 (558h52m13.349802211s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.479894605 (558h52m13.35097765s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.481013675 (558h52m13.35209671s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.482208331 (558h52m13.353291366s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:14.924931350 (558h52m14.796014395s) [upload=489964]: Next returned nil segment Dec 28 22:21:14.924963901 (558h52m14.796046966s) [upload=489964]: Waiting for error group managing segments... Dec 28 22:21:15.127660355 (558h52m14.99874339s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 28 22:21:15.345729562 (558h52m15.216812607s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.353583838 (558h52m15.224666873s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.491284908 (558h52m15.362367943s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.516145146 (558h52m15.387228191s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.539545846 (558h52m15.410628901s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.549881166 (558h52m15.420964201s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.595277810 (558h52m15.466360855s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.246156712 (558h52m16.117239757s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.275617411 (558h52m16.146700456s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.309349274 (558h52m16.180432319s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.342178146 (558h52m16.213261191s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.377649129 (558h52m16.248732164s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.430356415 (558h52m16.30143945s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559330918 (558h52m16.430413953s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559353802 (558h52m16.430436847s) [seg_pos={0 0}, upload=489964]: Segment reached optimal threshold of 14 pieces. Dec 28 22:21:16.559599279 (558h52m16.430682314s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559607655 (558h52m16.4306907s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559789661 (558h52m16.430872706s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559810030 (558h52m16.430893065s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559920550 (558h52m16.431003596s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559930830 (558h52m16.431013875s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559951920 (558h52m16.431034965s) [seg_pos={0 0}, upload=489964]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:21:16.560031201 (558h52m16.431114236s) [upload=489964]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:21:16.560037152 (558h52m16.431120187s) [upload=489964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:21:16.847427049 (558h52m16.718510084s) [upload=489964]: Done uploading segments... Dec 28 22:21:16.847446086 (558h52m16.718529131s) [upload=489964]: Upload finished. err: <nil> Duration: 19.51379514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X....................X.....X..] duration: 6.022974051s trace: https://jaeger.dev.storj.io/trace/5e4656c542eca88e FAIL (2025-12-28 22:35:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 4LKIIYWMQ4RTMQVFD3URYEZ4EFLGY6WCRYOMW6AUV6OIYGTJZLEQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: 57FEQYPP7I33MP3HRYFV22BOJ7MNM5TFDN3GRRDIQJ7MUFZR3MJQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: FOWBW36IZCJ3IJYITKT4VKQBR6SVPXY4SOTV3WC5NTHZL74Q4I3A) context deadline exceeded ======LOGS====== Dec 28 22:33:29.729946652 (559h4m29.601029697s) [upload=490140]: Starting upload Dec 28 22:33:29.730011725 (559h4m29.60109476s) [upload=490140]: Done starting upload Dec 28 22:33:29.730059898 (559h4m29.601142933s) [upload=490140]: Uploading segments... Dec 28 22:33:29.730191068 (559h4m29.601274103s) [upload=490140]: Got next segment. Inline: false Dec 28 22:33:29.730225061 (559h4m29.601308096s) [upload=490140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:33:29.730229050 (559h4m29.601312085s) [upload=490140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:33:29.982960409 (559h4m29.854043444s) [seg_pos={0 0}, upload=490140]: Begin upload segment... Dec 28 22:33:29.983015174 (559h4m29.854098209s) [seg_pos={0 0}, upload=490140]: Done begin upload segment. Dec 28 22:33:29.983073906 (559h4m29.854156941s) [noise=true , address=145.79.227.51:4409 , piece_id=ZWRDHUVEM6GHIJHV3RUJO5XJJCZ3ZQTOU5HAK2URT2HK5I5ZXZKA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.983088052 (559h4m29.854171097s) [noise=true , address=145.79.227.26:4654 , piece_id=YTXOCCTG3AVOUZDFPOXEKBZXIOZPCQJ4VSSG7HHQWVT52SZCCGJA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.983688816 (559h4m29.854771851s) [noise=true , address=145.79.227.38:4979 , piece_id=TFPL7BNNFWOGW6ZB4AE5437CZ2U7TCD4SSQHTMJ4PU27B2RDFS3Q, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.984142400 (559h4m29.855225445s) [noise=true , address=145.79.227.39:4015 , piece_id=57FEQYPP7I33MP3HRYFV22BOJ7MNM5TFDN3GRRDIQJ7MUFZR3MJQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.984532923 (559h4m29.855615958s) [noise=true , address=45.91.116.8:4045 , piece_id=NFOF2OA7JLTOVAHY4BC54OKNACCNP36U6DPDQWNYRNX3NK7JZSZA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.984876387 (559h4m29.855959422s) [noise=true , address=45.91.116.7:4012 , piece_id=FOWBW36IZCJ3IJYITKT4VKQBR6SVPXY4SOTV3WC5NTHZL74Q4I3A, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.985415613 (559h4m29.856498659s) [noise=true , address=145.79.227.15:4875 , piece_id=JWMDBQGJY2LTQGCBYOXMFF3FM7SIYR6W6LCPETCRWIOOXNXZYWPA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.985952105 (559h4m29.85703515s) [noise=true , address=45.91.116.12:4225 , piece_id=4LKIIYWMQ4RTMQVFD3URYEZ4EFLGY6WCRYOMW6AUV6OIYGTJZLEQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.986963731 (559h4m29.858046766s) [noise=true , address=145.79.227.19:4078 , piece_id=LQWLLQ2OIXPJHB6WTNAWPNLBIDTZEGSCHMX5DSVCCXRB4YJMJPXQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.987187446 (559h4m29.858270482s) [noise=true , address=45.91.116.6:4974 , piece_id=6UFJLFBEKPNQ7U47GD5XRP4L47HO3DJPMAPRLFXTJXK47KIBTBAQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.988386640 (559h4m29.859469676s) [noise=true , address=145.79.227.50:4374 , piece_id=DQCLZ5UZUKXHYK5MNWC7GBCR6RXAYOUFWGKFSSTBYQSVAS7MR47A, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.988736587 (559h4m29.859819632s) [noise=true , address=45.91.116.9:4261 , piece_id=YWFQCOVU6G2MVFYNPUL37MXC5TEOOHY3PZH3RM5RMTBVQRLIABDQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.989595411 (559h4m29.860678447s) [noise=true , address=145.79.227.48:4301 , piece_id=2FNMGYKOCOSK7OCNSKSLPX6GIRT6PFIL7VI6FTWR5WEALWYRY4RQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.990731414 (559h4m29.861814459s) [noise=true , address=145.79.227.37:4566 , piece_id=UVVMZOAZWTONSFF5ODSBJ3UZB2NMJMCI7VK76A24HIOFHQ5X3WQA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.993898394 (559h4m29.86498143s) [noise=true , address=77.90.31.21:4878 , piece_id=TBJZYLI7R2CE7JY7GPDTPXDITWAZIVC2TLJ6XESE6OZ343OZKWXA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:30.000210902 (559h4m29.871293957s) [noise=true , address=145.79.227.46:4601 , piece_id=P6H3HYQHUEZZZHTSFCHV6CSJ2C75XEBRIT3EZDTDCCGJYLI5RRBA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:31.595685625 (559h4m31.466768671s) [upload=490140]: Next returned nil segment Dec 28 22:33:31.595703990 (559h4m31.466787035s) [upload=490140]: Waiting for error group managing segments... Dec 28 22:33:31.976431013 (559h4m31.847514058s) [noise=true , address=145.79.227.26:4654 , piece_id=YTXOCCTG3AVOUZDFPOXEKBZXIOZPCQJ4VSSG7HHQWVT52SZCCGJA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:31.992221107 (559h4m31.863304152s) [noise=true , address=145.79.227.51:4409 , piece_id=ZWRDHUVEM6GHIJHV3RUJO5XJJCZ3ZQTOU5HAK2URT2HK5I5ZXZKA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.000376273 (559h4m31.871459318s) [noise=true , address=145.79.227.19:4078 , piece_id=LQWLLQ2OIXPJHB6WTNAWPNLBIDTZEGSCHMX5DSVCCXRB4YJMJPXQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.002835345 (559h4m31.8739184s) [noise=true , address=145.79.227.37:4566 , piece_id=UVVMZOAZWTONSFF5ODSBJ3UZB2NMJMCI7VK76A24HIOFHQ5X3WQA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.004171389 (559h4m31.875254424s) [noise=true , address=145.79.227.39:4015 , piece_id=57FEQYPP7I33MP3HRYFV22BOJ7MNM5TFDN3GRRDIQJ7MUFZR3MJQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.007413442 (559h4m31.878496487s) [noise=true , address=145.79.227.50:4374 , piece_id=DQCLZ5UZUKXHYK5MNWC7GBCR6RXAYOUFWGKFSSTBYQSVAS7MR47A, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.014865720 (559h4m31.885948755s) [noise=true , address=145.79.227.38:4979 , piece_id=TFPL7BNNFWOGW6ZB4AE5437CZ2U7TCD4SSQHTMJ4PU27B2RDFS3Q, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.017462344 (559h4m31.888545379s) [noise=true , address=145.79.227.46:4601 , piece_id=P6H3HYQHUEZZZHTSFCHV6CSJ2C75XEBRIT3EZDTDCCGJYLI5RRBA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.018098896 (559h4m31.889181941s) [noise=true , address=145.79.227.15:4875 , piece_id=JWMDBQGJY2LTQGCBYOXMFF3FM7SIYR6W6LCPETCRWIOOXNXZYWPA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.020611821 (559h4m31.891694856s) [noise=true , address=145.79.227.48:4301 , piece_id=2FNMGYKOCOSK7OCNSKSLPX6GIRT6PFIL7VI6FTWR5WEALWYRY4RQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.224145070 (559h4m32.095228106s) [noise=true , address=45.91.116.12:4225 , piece_id=4LKIIYWMQ4RTMQVFD3URYEZ4EFLGY6WCRYOMW6AUV6OIYGTJZLEQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.259872001 (559h4m32.130955046s) [noise=true , address=77.90.31.21:4878 , piece_id=TBJZYLI7R2CE7JY7GPDTPXDITWAZIVC2TLJ6XESE6OZ343OZKWXA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.272662344 (559h4m32.143745389s) [noise=true , address=45.91.116.9:4261 , piece_id=YWFQCOVU6G2MVFYNPUL37MXC5TEOOHY3PZH3RM5RMTBVQRLIABDQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.272684415 (559h4m32.14376745s) [seg_pos={0 0}, upload=490140]: Segment reached optimal threshold of 13 pieces. Dec 28 22:33:32.272850472 (559h4m32.143933507s) [noise=true , address=45.91.116.6:4974 , piece_id=6UFJLFBEKPNQ7U47GD5XRP4L47HO3DJPMAPRLFXTJXK47KIBTBAQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:44822->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:44822->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44822->45.91.116.6:4974: use of closed network connection Dec 28 22:33:32.272856653 (559h4m32.143939688s) [noise=true , address=45.91.116.6:4974 , piece_id=6UFJLFBEKPNQ7U47GD5XRP4L47HO3DJPMAPRLFXTJXK47KIBTBAQ, seg_pos={0 0}, upload=490140]: Long tail cancellation due to optimal threshold Dec 28 22:33:32.273054170 (559h4m32.144137205s) [noise=true , address=45.91.116.7:4012 , piece_id=FOWBW36IZCJ3IJYITKT4VKQBR6SVPXY4SOTV3WC5NTHZL74Q4I3A, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.273116308 (559h4m32.144199353s) [noise=true , address=45.91.116.8:4045 , piece_id=NFOF2OA7JLTOVAHY4BC54OKNACCNP36U6DPDQWNYRNX3NK7JZSZA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 22:33:32.273123332 (559h4m32.144206377s) [noise=true , address=45.91.116.8:4045 , piece_id=NFOF2OA7JLTOVAHY4BC54OKNACCNP36U6DPDQWNYRNX3NK7JZSZA, seg_pos={0 0}, upload=490140]: Long tail cancellation due to optimal threshold Dec 28 22:33:32.273145975 (559h4m32.14422901s) [seg_pos={0 0}, upload=490140]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 28 22:33:32.273254051 (559h4m32.144337086s) [upload=490140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:33:32.273259291 (559h4m32.144342326s) [upload=490140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:33:32.568117145 (559h4m32.43920019s) [upload=490140]: Done uploading segments... Dec 28 22:33:32.568136302 (559h4m32.439219347s) [upload=490140]: Upload finished. err: <nil> Duration: 6.023008267s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.X..................] duration: 29.330970115s trace: https://jaeger.dev.storj.io/trace/2dde5fb84d6f9aef FAIL (2025-12-28 22:35:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: 7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA) context deadline exceeded; piecestore: (Node ID: 1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, Piece ID: TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 28 22:21:13.466126077 (558h52m13.337209112s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466130635 (558h52m13.33721367s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466584950 (558h52m13.337667985s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466681324 (558h52m13.337764369s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466954685 (558h52m13.33803772s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467283190 (558h52m13.338366235s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467651912 (558h52m13.338734957s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467926144 (558h52m13.33900918s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468224863 (558h52m13.339307908s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468533250 (558h52m13.339616295s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468852889 (558h52m13.339935924s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.474952334 (558h52m13.346035369s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.476142490 (558h52m13.347225525s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.477282110 (558h52m13.348365145s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.478719176 (558h52m13.349802211s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.479894605 (558h52m13.35097765s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.481013675 (558h52m13.35209671s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.482208331 (558h52m13.353291366s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:14.924931350 (558h52m14.796014395s) [upload=489964]: Next returned nil segment Dec 28 22:21:14.924963901 (558h52m14.796046966s) [upload=489964]: Waiting for error group managing segments... Dec 28 22:21:15.127660355 (558h52m14.99874339s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 28 22:21:15.345729562 (558h52m15.216812607s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.353583838 (558h52m15.224666873s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.491284908 (558h52m15.362367943s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.516145146 (558h52m15.387228191s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.539545846 (558h52m15.410628901s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.549881166 (558h52m15.420964201s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.595277810 (558h52m15.466360855s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.246156712 (558h52m16.117239757s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.275617411 (558h52m16.146700456s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.309349274 (558h52m16.180432319s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.342178146 (558h52m16.213261191s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.377649129 (558h52m16.248732164s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.430356415 (558h52m16.30143945s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559330918 (558h52m16.430413953s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559353802 (558h52m16.430436847s) [seg_pos={0 0}, upload=489964]: Segment reached optimal threshold of 14 pieces. Dec 28 22:21:16.559599279 (558h52m16.430682314s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559607655 (558h52m16.4306907s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559789661 (558h52m16.430872706s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559810030 (558h52m16.430893065s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559920550 (558h52m16.431003596s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559930830 (558h52m16.431013875s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559951920 (558h52m16.431034965s) [seg_pos={0 0}, upload=489964]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:21:16.560031201 (558h52m16.431114236s) [upload=489964]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:21:16.560037152 (558h52m16.431120187s) [upload=489964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:21:16.847427049 (558h52m16.718510084s) [upload=489964]: Done uploading segments... Dec 28 22:21:16.847446086 (558h52m16.718529131s) [upload=489964]: Upload finished. err: <nil> Duration: 29.331001125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X....................X.....X.] duration: 7.275633387s trace: https://jaeger.dev.storj.io/trace/5e4656c542eca88e FAIL (2025-12-28 22:34:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 4LKIIYWMQ4RTMQVFD3URYEZ4EFLGY6WCRYOMW6AUV6OIYGTJZLEQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: 57FEQYPP7I33MP3HRYFV22BOJ7MNM5TFDN3GRRDIQJ7MUFZR3MJQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: FOWBW36IZCJ3IJYITKT4VKQBR6SVPXY4SOTV3WC5NTHZL74Q4I3A) context deadline exceeded ======LOGS====== Dec 28 22:33:29.729946652 (559h4m29.601029697s) [upload=490140]: Starting upload Dec 28 22:33:29.730011725 (559h4m29.60109476s) [upload=490140]: Done starting upload Dec 28 22:33:29.730059898 (559h4m29.601142933s) [upload=490140]: Uploading segments... Dec 28 22:33:29.730191068 (559h4m29.601274103s) [upload=490140]: Got next segment. Inline: false Dec 28 22:33:29.730225061 (559h4m29.601308096s) [upload=490140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:33:29.730229050 (559h4m29.601312085s) [upload=490140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:33:29.982960409 (559h4m29.854043444s) [seg_pos={0 0}, upload=490140]: Begin upload segment... Dec 28 22:33:29.983015174 (559h4m29.854098209s) [seg_pos={0 0}, upload=490140]: Done begin upload segment. Dec 28 22:33:29.983073906 (559h4m29.854156941s) [noise=true , address=145.79.227.51:4409 , piece_id=ZWRDHUVEM6GHIJHV3RUJO5XJJCZ3ZQTOU5HAK2URT2HK5I5ZXZKA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.983088052 (559h4m29.854171097s) [noise=true , address=145.79.227.26:4654 , piece_id=YTXOCCTG3AVOUZDFPOXEKBZXIOZPCQJ4VSSG7HHQWVT52SZCCGJA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.983688816 (559h4m29.854771851s) [noise=true , address=145.79.227.38:4979 , piece_id=TFPL7BNNFWOGW6ZB4AE5437CZ2U7TCD4SSQHTMJ4PU27B2RDFS3Q, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.984142400 (559h4m29.855225445s) [noise=true , address=145.79.227.39:4015 , piece_id=57FEQYPP7I33MP3HRYFV22BOJ7MNM5TFDN3GRRDIQJ7MUFZR3MJQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.984532923 (559h4m29.855615958s) [noise=true , address=45.91.116.8:4045 , piece_id=NFOF2OA7JLTOVAHY4BC54OKNACCNP36U6DPDQWNYRNX3NK7JZSZA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.984876387 (559h4m29.855959422s) [noise=true , address=45.91.116.7:4012 , piece_id=FOWBW36IZCJ3IJYITKT4VKQBR6SVPXY4SOTV3WC5NTHZL74Q4I3A, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.985415613 (559h4m29.856498659s) [noise=true , address=145.79.227.15:4875 , piece_id=JWMDBQGJY2LTQGCBYOXMFF3FM7SIYR6W6LCPETCRWIOOXNXZYWPA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.985952105 (559h4m29.85703515s) [noise=true , address=45.91.116.12:4225 , piece_id=4LKIIYWMQ4RTMQVFD3URYEZ4EFLGY6WCRYOMW6AUV6OIYGTJZLEQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.986963731 (559h4m29.858046766s) [noise=true , address=145.79.227.19:4078 , piece_id=LQWLLQ2OIXPJHB6WTNAWPNLBIDTZEGSCHMX5DSVCCXRB4YJMJPXQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.987187446 (559h4m29.858270482s) [noise=true , address=45.91.116.6:4974 , piece_id=6UFJLFBEKPNQ7U47GD5XRP4L47HO3DJPMAPRLFXTJXK47KIBTBAQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.988386640 (559h4m29.859469676s) [noise=true , address=145.79.227.50:4374 , piece_id=DQCLZ5UZUKXHYK5MNWC7GBCR6RXAYOUFWGKFSSTBYQSVAS7MR47A, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.988736587 (559h4m29.859819632s) [noise=true , address=45.91.116.9:4261 , piece_id=YWFQCOVU6G2MVFYNPUL37MXC5TEOOHY3PZH3RM5RMTBVQRLIABDQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.989595411 (559h4m29.860678447s) [noise=true , address=145.79.227.48:4301 , piece_id=2FNMGYKOCOSK7OCNSKSLPX6GIRT6PFIL7VI6FTWR5WEALWYRY4RQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.990731414 (559h4m29.861814459s) [noise=true , address=145.79.227.37:4566 , piece_id=UVVMZOAZWTONSFF5ODSBJ3UZB2NMJMCI7VK76A24HIOFHQ5X3WQA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.993898394 (559h4m29.86498143s) [noise=true , address=77.90.31.21:4878 , piece_id=TBJZYLI7R2CE7JY7GPDTPXDITWAZIVC2TLJ6XESE6OZ343OZKWXA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:30.000210902 (559h4m29.871293957s) [noise=true , address=145.79.227.46:4601 , piece_id=P6H3HYQHUEZZZHTSFCHV6CSJ2C75XEBRIT3EZDTDCCGJYLI5RRBA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:31.595685625 (559h4m31.466768671s) [upload=490140]: Next returned nil segment Dec 28 22:33:31.595703990 (559h4m31.466787035s) [upload=490140]: Waiting for error group managing segments... Dec 28 22:33:31.976431013 (559h4m31.847514058s) [noise=true , address=145.79.227.26:4654 , piece_id=YTXOCCTG3AVOUZDFPOXEKBZXIOZPCQJ4VSSG7HHQWVT52SZCCGJA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:31.992221107 (559h4m31.863304152s) [noise=true , address=145.79.227.51:4409 , piece_id=ZWRDHUVEM6GHIJHV3RUJO5XJJCZ3ZQTOU5HAK2URT2HK5I5ZXZKA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.000376273 (559h4m31.871459318s) [noise=true , address=145.79.227.19:4078 , piece_id=LQWLLQ2OIXPJHB6WTNAWPNLBIDTZEGSCHMX5DSVCCXRB4YJMJPXQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.002835345 (559h4m31.8739184s) [noise=true , address=145.79.227.37:4566 , piece_id=UVVMZOAZWTONSFF5ODSBJ3UZB2NMJMCI7VK76A24HIOFHQ5X3WQA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.004171389 (559h4m31.875254424s) [noise=true , address=145.79.227.39:4015 , piece_id=57FEQYPP7I33MP3HRYFV22BOJ7MNM5TFDN3GRRDIQJ7MUFZR3MJQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.007413442 (559h4m31.878496487s) [noise=true , address=145.79.227.50:4374 , piece_id=DQCLZ5UZUKXHYK5MNWC7GBCR6RXAYOUFWGKFSSTBYQSVAS7MR47A, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.014865720 (559h4m31.885948755s) [noise=true , address=145.79.227.38:4979 , piece_id=TFPL7BNNFWOGW6ZB4AE5437CZ2U7TCD4SSQHTMJ4PU27B2RDFS3Q, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.017462344 (559h4m31.888545379s) [noise=true , address=145.79.227.46:4601 , piece_id=P6H3HYQHUEZZZHTSFCHV6CSJ2C75XEBRIT3EZDTDCCGJYLI5RRBA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.018098896 (559h4m31.889181941s) [noise=true , address=145.79.227.15:4875 , piece_id=JWMDBQGJY2LTQGCBYOXMFF3FM7SIYR6W6LCPETCRWIOOXNXZYWPA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.020611821 (559h4m31.891694856s) [noise=true , address=145.79.227.48:4301 , piece_id=2FNMGYKOCOSK7OCNSKSLPX6GIRT6PFIL7VI6FTWR5WEALWYRY4RQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.224145070 (559h4m32.095228106s) [noise=true , address=45.91.116.12:4225 , piece_id=4LKIIYWMQ4RTMQVFD3URYEZ4EFLGY6WCRYOMW6AUV6OIYGTJZLEQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.259872001 (559h4m32.130955046s) [noise=true , address=77.90.31.21:4878 , piece_id=TBJZYLI7R2CE7JY7GPDTPXDITWAZIVC2TLJ6XESE6OZ343OZKWXA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.272662344 (559h4m32.143745389s) [noise=true , address=45.91.116.9:4261 , piece_id=YWFQCOVU6G2MVFYNPUL37MXC5TEOOHY3PZH3RM5RMTBVQRLIABDQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.272684415 (559h4m32.14376745s) [seg_pos={0 0}, upload=490140]: Segment reached optimal threshold of 13 pieces. Dec 28 22:33:32.272850472 (559h4m32.143933507s) [noise=true , address=45.91.116.6:4974 , piece_id=6UFJLFBEKPNQ7U47GD5XRP4L47HO3DJPMAPRLFXTJXK47KIBTBAQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:44822->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:44822->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44822->45.91.116.6:4974: use of closed network connection Dec 28 22:33:32.272856653 (559h4m32.143939688s) [noise=true , address=45.91.116.6:4974 , piece_id=6UFJLFBEKPNQ7U47GD5XRP4L47HO3DJPMAPRLFXTJXK47KIBTBAQ, seg_pos={0 0}, upload=490140]: Long tail cancellation due to optimal threshold Dec 28 22:33:32.273054170 (559h4m32.144137205s) [noise=true , address=45.91.116.7:4012 , piece_id=FOWBW36IZCJ3IJYITKT4VKQBR6SVPXY4SOTV3WC5NTHZL74Q4I3A, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.273116308 (559h4m32.144199353s) [noise=true , address=45.91.116.8:4045 , piece_id=NFOF2OA7JLTOVAHY4BC54OKNACCNP36U6DPDQWNYRNX3NK7JZSZA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 22:33:32.273123332 (559h4m32.144206377s) [noise=true , address=45.91.116.8:4045 , piece_id=NFOF2OA7JLTOVAHY4BC54OKNACCNP36U6DPDQWNYRNX3NK7JZSZA, seg_pos={0 0}, upload=490140]: Long tail cancellation due to optimal threshold Dec 28 22:33:32.273145975 (559h4m32.14422901s) [seg_pos={0 0}, upload=490140]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 28 22:33:32.273254051 (559h4m32.144337086s) [upload=490140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:33:32.273259291 (559h4m32.144342326s) [upload=490140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:33:32.568117145 (559h4m32.43920019s) [upload=490140]: Done uploading segments... Dec 28 22:33:32.568136302 (559h4m32.439219347s) [upload=490140]: Upload finished. err: <nil> Duration: 7.275664517s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.X.................] duration: 19.647756198s trace: https://jaeger.dev.storj.io/trace/2dde5fb84d6f9aef FAIL (2025-12-28 22:34:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: 7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA) context deadline exceeded; piecestore: (Node ID: 1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, Piece ID: TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 28 22:21:13.466126077 (558h52m13.337209112s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466130635 (558h52m13.33721367s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466584950 (558h52m13.337667985s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466681324 (558h52m13.337764369s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466954685 (558h52m13.33803772s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467283190 (558h52m13.338366235s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467651912 (558h52m13.338734957s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467926144 (558h52m13.33900918s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468224863 (558h52m13.339307908s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468533250 (558h52m13.339616295s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468852889 (558h52m13.339935924s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.474952334 (558h52m13.346035369s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.476142490 (558h52m13.347225525s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.477282110 (558h52m13.348365145s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.478719176 (558h52m13.349802211s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.479894605 (558h52m13.35097765s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.481013675 (558h52m13.35209671s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.482208331 (558h52m13.353291366s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:14.924931350 (558h52m14.796014395s) [upload=489964]: Next returned nil segment Dec 28 22:21:14.924963901 (558h52m14.796046966s) [upload=489964]: Waiting for error group managing segments... Dec 28 22:21:15.127660355 (558h52m14.99874339s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 28 22:21:15.345729562 (558h52m15.216812607s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.353583838 (558h52m15.224666873s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.491284908 (558h52m15.362367943s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.516145146 (558h52m15.387228191s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.539545846 (558h52m15.410628901s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.549881166 (558h52m15.420964201s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.595277810 (558h52m15.466360855s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.246156712 (558h52m16.117239757s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.275617411 (558h52m16.146700456s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.309349274 (558h52m16.180432319s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.342178146 (558h52m16.213261191s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.377649129 (558h52m16.248732164s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.430356415 (558h52m16.30143945s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559330918 (558h52m16.430413953s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559353802 (558h52m16.430436847s) [seg_pos={0 0}, upload=489964]: Segment reached optimal threshold of 14 pieces. Dec 28 22:21:16.559599279 (558h52m16.430682314s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559607655 (558h52m16.4306907s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559789661 (558h52m16.430872706s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559810030 (558h52m16.430893065s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559920550 (558h52m16.431003596s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559930830 (558h52m16.431013875s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559951920 (558h52m16.431034965s) [seg_pos={0 0}, upload=489964]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:21:16.560031201 (558h52m16.431114236s) [upload=489964]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:21:16.560037152 (558h52m16.431120187s) [upload=489964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:21:16.847427049 (558h52m16.718510084s) [upload=489964]: Done uploading segments... Dec 28 22:21:16.847446086 (558h52m16.718529131s) [upload=489964]: Upload finished. err: <nil> Duration: 19.647788189s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.X................] duration: 15.114102818s trace: https://jaeger.dev.storj.io/trace/2dde5fb84d6f9aef FAIL (2025-12-28 22:34:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: 7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA) context deadline exceeded; piecestore: (Node ID: 1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, Piece ID: TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 28 22:21:13.466126077 (558h52m13.337209112s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466130635 (558h52m13.33721367s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466584950 (558h52m13.337667985s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466681324 (558h52m13.337764369s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466954685 (558h52m13.33803772s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467283190 (558h52m13.338366235s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467651912 (558h52m13.338734957s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467926144 (558h52m13.33900918s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468224863 (558h52m13.339307908s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468533250 (558h52m13.339616295s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468852889 (558h52m13.339935924s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.474952334 (558h52m13.346035369s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.476142490 (558h52m13.347225525s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.477282110 (558h52m13.348365145s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.478719176 (558h52m13.349802211s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.479894605 (558h52m13.35097765s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.481013675 (558h52m13.35209671s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.482208331 (558h52m13.353291366s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:14.924931350 (558h52m14.796014395s) [upload=489964]: Next returned nil segment Dec 28 22:21:14.924963901 (558h52m14.796046966s) [upload=489964]: Waiting for error group managing segments... Dec 28 22:21:15.127660355 (558h52m14.99874339s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 28 22:21:15.345729562 (558h52m15.216812607s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.353583838 (558h52m15.224666873s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.491284908 (558h52m15.362367943s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.516145146 (558h52m15.387228191s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.539545846 (558h52m15.410628901s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.549881166 (558h52m15.420964201s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.595277810 (558h52m15.466360855s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.246156712 (558h52m16.117239757s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.275617411 (558h52m16.146700456s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.309349274 (558h52m16.180432319s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.342178146 (558h52m16.213261191s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.377649129 (558h52m16.248732164s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.430356415 (558h52m16.30143945s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559330918 (558h52m16.430413953s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559353802 (558h52m16.430436847s) [seg_pos={0 0}, upload=489964]: Segment reached optimal threshold of 14 pieces. Dec 28 22:21:16.559599279 (558h52m16.430682314s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559607655 (558h52m16.4306907s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559789661 (558h52m16.430872706s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559810030 (558h52m16.430893065s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559920550 (558h52m16.431003596s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559930830 (558h52m16.431013875s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559951920 (558h52m16.431034965s) [seg_pos={0 0}, upload=489964]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:21:16.560031201 (558h52m16.431114236s) [upload=489964]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:21:16.560037152 (558h52m16.431120187s) [upload=489964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:21:16.847427049 (558h52m16.718510084s) [upload=489964]: Done uploading segments... Dec 28 22:21:16.847446086 (558h52m16.718529131s) [upload=489964]: Upload finished. err: <nil> Duration: 15.114127264s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X....................X.....X] duration: 30.003720728s trace: https://jaeger.dev.storj.io/trace/5e4656c542eca88e FAIL (2025-12-28 22:33:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 4LKIIYWMQ4RTMQVFD3URYEZ4EFLGY6WCRYOMW6AUV6OIYGTJZLEQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: 57FEQYPP7I33MP3HRYFV22BOJ7MNM5TFDN3GRRDIQJ7MUFZR3MJQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: FOWBW36IZCJ3IJYITKT4VKQBR6SVPXY4SOTV3WC5NTHZL74Q4I3A) context deadline exceeded ======LOGS====== Dec 28 22:33:29.729946652 (559h4m29.601029697s) [upload=490140]: Starting upload Dec 28 22:33:29.730011725 (559h4m29.60109476s) [upload=490140]: Done starting upload Dec 28 22:33:29.730059898 (559h4m29.601142933s) [upload=490140]: Uploading segments... Dec 28 22:33:29.730191068 (559h4m29.601274103s) [upload=490140]: Got next segment. Inline: false Dec 28 22:33:29.730225061 (559h4m29.601308096s) [upload=490140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:33:29.730229050 (559h4m29.601312085s) [upload=490140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:33:29.982960409 (559h4m29.854043444s) [seg_pos={0 0}, upload=490140]: Begin upload segment... Dec 28 22:33:29.983015174 (559h4m29.854098209s) [seg_pos={0 0}, upload=490140]: Done begin upload segment. Dec 28 22:33:29.983073906 (559h4m29.854156941s) [noise=true , address=145.79.227.51:4409 , piece_id=ZWRDHUVEM6GHIJHV3RUJO5XJJCZ3ZQTOU5HAK2URT2HK5I5ZXZKA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.983088052 (559h4m29.854171097s) [noise=true , address=145.79.227.26:4654 , piece_id=YTXOCCTG3AVOUZDFPOXEKBZXIOZPCQJ4VSSG7HHQWVT52SZCCGJA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.983688816 (559h4m29.854771851s) [noise=true , address=145.79.227.38:4979 , piece_id=TFPL7BNNFWOGW6ZB4AE5437CZ2U7TCD4SSQHTMJ4PU27B2RDFS3Q, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.984142400 (559h4m29.855225445s) [noise=true , address=145.79.227.39:4015 , piece_id=57FEQYPP7I33MP3HRYFV22BOJ7MNM5TFDN3GRRDIQJ7MUFZR3MJQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.984532923 (559h4m29.855615958s) [noise=true , address=45.91.116.8:4045 , piece_id=NFOF2OA7JLTOVAHY4BC54OKNACCNP36U6DPDQWNYRNX3NK7JZSZA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.984876387 (559h4m29.855959422s) [noise=true , address=45.91.116.7:4012 , piece_id=FOWBW36IZCJ3IJYITKT4VKQBR6SVPXY4SOTV3WC5NTHZL74Q4I3A, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.985415613 (559h4m29.856498659s) [noise=true , address=145.79.227.15:4875 , piece_id=JWMDBQGJY2LTQGCBYOXMFF3FM7SIYR6W6LCPETCRWIOOXNXZYWPA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.985952105 (559h4m29.85703515s) [noise=true , address=45.91.116.12:4225 , piece_id=4LKIIYWMQ4RTMQVFD3URYEZ4EFLGY6WCRYOMW6AUV6OIYGTJZLEQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.986963731 (559h4m29.858046766s) [noise=true , address=145.79.227.19:4078 , piece_id=LQWLLQ2OIXPJHB6WTNAWPNLBIDTZEGSCHMX5DSVCCXRB4YJMJPXQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.987187446 (559h4m29.858270482s) [noise=true , address=45.91.116.6:4974 , piece_id=6UFJLFBEKPNQ7U47GD5XRP4L47HO3DJPMAPRLFXTJXK47KIBTBAQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.988386640 (559h4m29.859469676s) [noise=true , address=145.79.227.50:4374 , piece_id=DQCLZ5UZUKXHYK5MNWC7GBCR6RXAYOUFWGKFSSTBYQSVAS7MR47A, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.988736587 (559h4m29.859819632s) [noise=true , address=45.91.116.9:4261 , piece_id=YWFQCOVU6G2MVFYNPUL37MXC5TEOOHY3PZH3RM5RMTBVQRLIABDQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.989595411 (559h4m29.860678447s) [noise=true , address=145.79.227.48:4301 , piece_id=2FNMGYKOCOSK7OCNSKSLPX6GIRT6PFIL7VI6FTWR5WEALWYRY4RQ, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.990731414 (559h4m29.861814459s) [noise=true , address=145.79.227.37:4566 , piece_id=UVVMZOAZWTONSFF5ODSBJ3UZB2NMJMCI7VK76A24HIOFHQ5X3WQA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:29.993898394 (559h4m29.86498143s) [noise=true , address=77.90.31.21:4878 , piece_id=TBJZYLI7R2CE7JY7GPDTPXDITWAZIVC2TLJ6XESE6OZ343OZKWXA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:30.000210902 (559h4m29.871293957s) [noise=true , address=145.79.227.46:4601 , piece_id=P6H3HYQHUEZZZHTSFCHV6CSJ2C75XEBRIT3EZDTDCCGJYLI5RRBA, seg_pos={0 0}, upload=490140]: Uploading piece... Dec 28 22:33:31.595685625 (559h4m31.466768671s) [upload=490140]: Next returned nil segment Dec 28 22:33:31.595703990 (559h4m31.466787035s) [upload=490140]: Waiting for error group managing segments... Dec 28 22:33:31.976431013 (559h4m31.847514058s) [noise=true , address=145.79.227.26:4654 , piece_id=YTXOCCTG3AVOUZDFPOXEKBZXIOZPCQJ4VSSG7HHQWVT52SZCCGJA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:31.992221107 (559h4m31.863304152s) [noise=true , address=145.79.227.51:4409 , piece_id=ZWRDHUVEM6GHIJHV3RUJO5XJJCZ3ZQTOU5HAK2URT2HK5I5ZXZKA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.000376273 (559h4m31.871459318s) [noise=true , address=145.79.227.19:4078 , piece_id=LQWLLQ2OIXPJHB6WTNAWPNLBIDTZEGSCHMX5DSVCCXRB4YJMJPXQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.002835345 (559h4m31.8739184s) [noise=true , address=145.79.227.37:4566 , piece_id=UVVMZOAZWTONSFF5ODSBJ3UZB2NMJMCI7VK76A24HIOFHQ5X3WQA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.004171389 (559h4m31.875254424s) [noise=true , address=145.79.227.39:4015 , piece_id=57FEQYPP7I33MP3HRYFV22BOJ7MNM5TFDN3GRRDIQJ7MUFZR3MJQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.007413442 (559h4m31.878496487s) [noise=true , address=145.79.227.50:4374 , piece_id=DQCLZ5UZUKXHYK5MNWC7GBCR6RXAYOUFWGKFSSTBYQSVAS7MR47A, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.014865720 (559h4m31.885948755s) [noise=true , address=145.79.227.38:4979 , piece_id=TFPL7BNNFWOGW6ZB4AE5437CZ2U7TCD4SSQHTMJ4PU27B2RDFS3Q, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.017462344 (559h4m31.888545379s) [noise=true , address=145.79.227.46:4601 , piece_id=P6H3HYQHUEZZZHTSFCHV6CSJ2C75XEBRIT3EZDTDCCGJYLI5RRBA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.018098896 (559h4m31.889181941s) [noise=true , address=145.79.227.15:4875 , piece_id=JWMDBQGJY2LTQGCBYOXMFF3FM7SIYR6W6LCPETCRWIOOXNXZYWPA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.020611821 (559h4m31.891694856s) [noise=true , address=145.79.227.48:4301 , piece_id=2FNMGYKOCOSK7OCNSKSLPX6GIRT6PFIL7VI6FTWR5WEALWYRY4RQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.224145070 (559h4m32.095228106s) [noise=true , address=45.91.116.12:4225 , piece_id=4LKIIYWMQ4RTMQVFD3URYEZ4EFLGY6WCRYOMW6AUV6OIYGTJZLEQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.259872001 (559h4m32.130955046s) [noise=true , address=77.90.31.21:4878 , piece_id=TBJZYLI7R2CE7JY7GPDTPXDITWAZIVC2TLJ6XESE6OZ343OZKWXA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.272662344 (559h4m32.143745389s) [noise=true , address=45.91.116.9:4261 , piece_id=YWFQCOVU6G2MVFYNPUL37MXC5TEOOHY3PZH3RM5RMTBVQRLIABDQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.272684415 (559h4m32.14376745s) [seg_pos={0 0}, upload=490140]: Segment reached optimal threshold of 13 pieces. Dec 28 22:33:32.272850472 (559h4m32.143933507s) [noise=true , address=45.91.116.6:4974 , piece_id=6UFJLFBEKPNQ7U47GD5XRP4L47HO3DJPMAPRLFXTJXK47KIBTBAQ, seg_pos={0 0}, upload=490140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:44822->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:44822->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44822->45.91.116.6:4974: use of closed network connection Dec 28 22:33:32.272856653 (559h4m32.143939688s) [noise=true , address=45.91.116.6:4974 , piece_id=6UFJLFBEKPNQ7U47GD5XRP4L47HO3DJPMAPRLFXTJXK47KIBTBAQ, seg_pos={0 0}, upload=490140]: Long tail cancellation due to optimal threshold Dec 28 22:33:32.273054170 (559h4m32.144137205s) [noise=true , address=45.91.116.7:4012 , piece_id=FOWBW36IZCJ3IJYITKT4VKQBR6SVPXY4SOTV3WC5NTHZL74Q4I3A, seg_pos={0 0}, upload=490140]: Done uploading piece. err: <nil> Dec 28 22:33:32.273116308 (559h4m32.144199353s) [noise=true , address=45.91.116.8:4045 , piece_id=NFOF2OA7JLTOVAHY4BC54OKNACCNP36U6DPDQWNYRNX3NK7JZSZA, seg_pos={0 0}, upload=490140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 22:33:32.273123332 (559h4m32.144206377s) [noise=true , address=45.91.116.8:4045 , piece_id=NFOF2OA7JLTOVAHY4BC54OKNACCNP36U6DPDQWNYRNX3NK7JZSZA, seg_pos={0 0}, upload=490140]: Long tail cancellation due to optimal threshold Dec 28 22:33:32.273145975 (559h4m32.14422901s) [seg_pos={0 0}, upload=490140]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 28 22:33:32.273254051 (559h4m32.144337086s) [upload=490140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:33:32.273259291 (559h4m32.144342326s) [upload=490140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:33:32.568117145 (559h4m32.43920019s) [upload=490140]: Done uploading segments... Dec 28 22:33:32.568136302 (559h4m32.439219347s) [upload=490140]: Upload finished. err: <nil> Duration: 30.003743061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X....................X.....] duration: 28.307445512s trace: https://jaeger.dev.storj.io/trace/46050a20bb3702b9 FAIL (2025-12-28 22:33:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 7LQPQ4BBVDLU7R5E2CJ6OVWJNXP6OWMREW77JCTKETK6GRHEKWFQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 6LK46DLK3C7A5HKHXBI4H6P2RF57HRAEONE744K62IIWIKFRFEJA) context deadline exceeded ======LOGS====== Dec 28 22:29:03.234193464 (559h0m3.105276499s) [upload=490074]: Starting upload Dec 28 22:29:03.234253228 (559h0m3.105336274s) [upload=490074]: Done starting upload Dec 28 22:29:03.234343239 (559h0m3.105426284s) [upload=490074]: Uploading segments... Dec 28 22:29:03.234476433 (559h0m3.105559468s) [upload=490074]: Got next segment. Inline: false Dec 28 22:29:03.234539113 (559h0m3.105622148s) [upload=490074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:29:03.234545825 (559h0m3.10562887s) [upload=490074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:29:03.480992019 (559h0m3.352075054s) [seg_pos={0 0}, upload=490074]: Begin upload segment... Dec 28 22:29:03.481045751 (559h0m3.352128786s) [seg_pos={0 0}, upload=490074]: Done begin upload segment. Dec 28 22:29:03.481090216 (559h0m3.352173251s) [noise=true , address=145.79.227.6:4259 , piece_id=6TDR5BWNEXSSPZRWMU2KNALEQGQTHCNQB5CWRX3XQHN2VTCE2EOA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.481132456 (559h0m3.352215491s) [noise=true , address=145.79.227.48:4302 , piece_id=OYVDCHEUYD37EK5MN3WAWH3LGCNL5T454JU5EDS4LHYESGAEWNWQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.481935585 (559h0m3.35301862s) [noise=true , address=45.91.116.4:4940 , piece_id=J226BBVOC5OOL7GFBDQYKPMAXYLR54ZNI6L2JPFGDVN6TLLX4VAA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.482331749 (559h0m3.353414794s) [noise=true , address=45.91.116.5:4297 , piece_id=HAI53YOMCKCJG5ZYNK3N4YAQSKLKB6KUAAKV34LAZKLDV2WJ4LGQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.482693668 (559h0m3.353776703s) [noise=true , address=145.79.227.50:4373 , piece_id=JQJXC3LNTT6XNQ3EQ5LBJQ2YP2LHPJNFQOHZH55IN6AXWDIBLXPQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.482820961 (559h0m3.353904006s) [noise=true , address=145.79.227.46:4602 , piece_id=7IHLOAS4LJUBJAV6CTXOLY5EVKDUU2KBI3I5UIVXMFMTSUHEBGYQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.483191526 (559h0m3.354274571s) [noise=true , address=145.79.227.7:4295 , piece_id=OZHNR5LJIEGABZSPP24BLKN2M5X2XUOODSMBBSF2GBA7YO6FFWEQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.483565929 (559h0m3.354648974s) [noise=true , address=145.79.227.47:4266 , piece_id=J6WFMSNTAC6REAJKINWXPIFPJMEKAG424GGK6CEORFU6TSFRJ5SA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.484064268 (559h0m3.355147313s) [noise=true , address=145.79.227.32:4834 , piece_id=R4PJDFDPLIOLMRMY4PMBJBGYNVCALBWQ6J75CS2ZIT4FZO4WEAKQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.484374909 (559h0m3.355457954s) [noise=true , address=45.91.116.13:4154 , piece_id=7LQPQ4BBVDLU7R5E2CJ6OVWJNXP6OWMREW77JCTKETK6GRHEKWFQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.484732259 (559h0m3.355815294s) [noise=true , address=145.79.227.44:4195 , piece_id=RSES7RLUFO3TXCAAT4LY2TK7VSE2BMXLPRNIUMIJGBQIH7D7VJAQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.485040046 (559h0m3.356123091s) [noise=true , address=145.79.227.52:4446 , piece_id=VIYSUY5RWVU4RKSRHH6XJV6XQCNSCXETE62DQRGUEFPRAAVMKLJQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.486251753 (559h0m3.357334798s) [noise=true , address=77.90.31.10:4518 , piece_id=R4L7ZIUOTVNT7QHFSAIXWDVU2IYC3EW7HSRVMECFKNGQSOISLXMA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.486794666 (559h0m3.357877701s) [noise=true , address=45.91.116.11:4117 , piece_id=6LK46DLK3C7A5HKHXBI4H6P2RF57HRAEONE744K62IIWIKFRFEJA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.487810641 (559h0m3.358893676s) [noise=true , address=77.90.31.11:4556 , piece_id=HI7EPN5GKQBSWU4LNSUEY76LMXPDYLG2Z5SQRMCHCZ6BYHZS34ZQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.488808671 (559h0m3.359891706s) [noise=true , address=145.79.227.5:4221 , piece_id=ZNKFRVH4FKX2OHYY7TFNXY7BL3ZCHGLEI4H3CPNN4TAIYLYEDZEA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:05.215604138 (559h0m5.086687173s) [upload=490074]: Next returned nil segment Dec 28 22:29:05.215618124 (559h0m5.086701159s) [upload=490074]: Waiting for error group managing segments... Dec 28 22:29:05.499544927 (559h0m5.370627963s) [noise=true , address=145.79.227.50:4373 , piece_id=JQJXC3LNTT6XNQ3EQ5LBJQ2YP2LHPJNFQOHZH55IN6AXWDIBLXPQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.508202160 (559h0m5.379285195s) [noise=true , address=145.79.227.52:4446 , piece_id=VIYSUY5RWVU4RKSRHH6XJV6XQCNSCXETE62DQRGUEFPRAAVMKLJQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.512343334 (559h0m5.383426369s) [noise=true , address=145.79.227.46:4602 , piece_id=7IHLOAS4LJUBJAV6CTXOLY5EVKDUU2KBI3I5UIVXMFMTSUHEBGYQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.512693731 (559h0m5.383776776s) [noise=true , address=145.79.227.47:4266 , piece_id=J6WFMSNTAC6REAJKINWXPIFPJMEKAG424GGK6CEORFU6TSFRJ5SA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.520833699 (559h0m5.391916745s) [noise=true , address=145.79.227.6:4259 , piece_id=6TDR5BWNEXSSPZRWMU2KNALEQGQTHCNQB5CWRX3XQHN2VTCE2EOA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.528284415 (559h0m5.39936745s) [noise=true , address=145.79.227.5:4221 , piece_id=ZNKFRVH4FKX2OHYY7TFNXY7BL3ZCHGLEI4H3CPNN4TAIYLYEDZEA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.535748456 (559h0m5.406831491s) [noise=true , address=145.79.227.48:4302 , piece_id=OYVDCHEUYD37EK5MN3WAWH3LGCNL5T454JU5EDS4LHYESGAEWNWQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.536951637 (559h0m5.408034682s) [noise=true , address=145.79.227.7:4295 , piece_id=OZHNR5LJIEGABZSPP24BLKN2M5X2XUOODSMBBSF2GBA7YO6FFWEQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.542701594 (559h0m5.41378463s) [noise=true , address=145.79.227.32:4834 , piece_id=R4PJDFDPLIOLMRMY4PMBJBGYNVCALBWQ6J75CS2ZIT4FZO4WEAKQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.734860740 (559h0m5.605943775s) [noise=true , address=77.90.31.11:4556 , piece_id=HI7EPN5GKQBSWU4LNSUEY76LMXPDYLG2Z5SQRMCHCZ6BYHZS34ZQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.738874121 (559h0m5.609957156s) [noise=true , address=45.91.116.11:4117 , piece_id=6LK46DLK3C7A5HKHXBI4H6P2RF57HRAEONE744K62IIWIKFRFEJA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.739730501 (559h0m5.610813536s) [noise=true , address=45.91.116.4:4940 , piece_id=J226BBVOC5OOL7GFBDQYKPMAXYLR54ZNI6L2JPFGDVN6TLLX4VAA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.758700138 (559h0m5.629783183s) [noise=true , address=45.91.116.13:4154 , piece_id=7LQPQ4BBVDLU7R5E2CJ6OVWJNXP6OWMREW77JCTKETK6GRHEKWFQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.758718813 (559h0m5.629801858s) [seg_pos={0 0}, upload=490074]: Segment reached optimal threshold of 13 pieces. Dec 28 22:29:05.758875382 (559h0m5.629958427s) [noise=true , address=77.90.31.10:4518 , piece_id=R4L7ZIUOTVNT7QHFSAIXWDVU2IYC3EW7HSRVMECFKNGQSOISLXMA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 28 22:29:05.758882054 (559h0m5.629965099s) [noise=true , address=77.90.31.10:4518 , piece_id=R4L7ZIUOTVNT7QHFSAIXWDVU2IYC3EW7HSRVMECFKNGQSOISLXMA, seg_pos={0 0}, upload=490074]: Long tail cancellation due to optimal threshold Dec 28 22:29:05.759057108 (559h0m5.630140183s) [noise=true , address=45.91.116.5:4297 , piece_id=HAI53YOMCKCJG5ZYNK3N4YAQSKLKB6KUAAKV34LAZKLDV2WJ4LGQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 22:29:05.759067457 (559h0m5.630150502s) [noise=true , address=45.91.116.5:4297 , piece_id=HAI53YOMCKCJG5ZYNK3N4YAQSKLKB6KUAAKV34LAZKLDV2WJ4LGQ, seg_pos={0 0}, upload=490074]: Long tail cancellation due to optimal threshold Dec 28 22:29:05.759353942 (559h0m5.630436997s) [noise=true , address=145.79.227.44:4195 , piece_id=RSES7RLUFO3TXCAAT4LY2TK7VSE2BMXLPRNIUMIJGBQIH7D7VJAQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 22:29:05.759365245 (559h0m5.63044828s) [noise=true , address=145.79.227.44:4195 , piece_id=RSES7RLUFO3TXCAAT4LY2TK7VSE2BMXLPRNIUMIJGBQIH7D7VJAQ, seg_pos={0 0}, upload=490074]: Long tail cancellation due to optimal threshold Dec 28 22:29:05.759389921 (559h0m5.630472966s) [seg_pos={0 0}, upload=490074]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:29:05.759472529 (559h0m5.630555574s) [upload=490074]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:29:05.759479372 (559h0m5.630562407s) [upload=490074]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:29:06.046694816 (559h0m5.917777851s) [upload=490074]: Done uploading segments... Dec 28 22:29:06.046712911 (559h0m5.917795946s) [upload=490074]: Upload finished. err: <nil> Duration: 28.307471302s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.X...............] duration: 22.335896696s trace: https://jaeger.dev.storj.io/trace/2dde5fb84d6f9aef FAIL (2025-12-28 22:32:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: 7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA) context deadline exceeded; piecestore: (Node ID: 1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, Piece ID: TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 28 22:21:13.466126077 (558h52m13.337209112s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466130635 (558h52m13.33721367s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466584950 (558h52m13.337667985s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466681324 (558h52m13.337764369s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466954685 (558h52m13.33803772s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467283190 (558h52m13.338366235s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467651912 (558h52m13.338734957s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467926144 (558h52m13.33900918s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468224863 (558h52m13.339307908s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468533250 (558h52m13.339616295s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468852889 (558h52m13.339935924s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.474952334 (558h52m13.346035369s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.476142490 (558h52m13.347225525s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.477282110 (558h52m13.348365145s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.478719176 (558h52m13.349802211s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.479894605 (558h52m13.35097765s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.481013675 (558h52m13.35209671s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.482208331 (558h52m13.353291366s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:14.924931350 (558h52m14.796014395s) [upload=489964]: Next returned nil segment Dec 28 22:21:14.924963901 (558h52m14.796046966s) [upload=489964]: Waiting for error group managing segments... Dec 28 22:21:15.127660355 (558h52m14.99874339s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 28 22:21:15.345729562 (558h52m15.216812607s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.353583838 (558h52m15.224666873s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.491284908 (558h52m15.362367943s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.516145146 (558h52m15.387228191s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.539545846 (558h52m15.410628901s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.549881166 (558h52m15.420964201s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.595277810 (558h52m15.466360855s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.246156712 (558h52m16.117239757s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.275617411 (558h52m16.146700456s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.309349274 (558h52m16.180432319s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.342178146 (558h52m16.213261191s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.377649129 (558h52m16.248732164s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.430356415 (558h52m16.30143945s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559330918 (558h52m16.430413953s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559353802 (558h52m16.430436847s) [seg_pos={0 0}, upload=489964]: Segment reached optimal threshold of 14 pieces. Dec 28 22:21:16.559599279 (558h52m16.430682314s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559607655 (558h52m16.4306907s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559789661 (558h52m16.430872706s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559810030 (558h52m16.430893065s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559920550 (558h52m16.431003596s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559930830 (558h52m16.431013875s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559951920 (558h52m16.431034965s) [seg_pos={0 0}, upload=489964]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:21:16.560031201 (558h52m16.431114236s) [upload=489964]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:21:16.560037152 (558h52m16.431120187s) [upload=489964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:21:16.847427049 (558h52m16.718510084s) [upload=489964]: Done uploading segments... Dec 28 22:21:16.847446086 (558h52m16.718529131s) [upload=489964]: Upload finished. err: <nil> Duration: 22.335932203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X....................X....] duration: 6.320277792s trace: https://jaeger.dev.storj.io/trace/46050a20bb3702b9 FAIL (2025-12-28 22:32:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 7LQPQ4BBVDLU7R5E2CJ6OVWJNXP6OWMREW77JCTKETK6GRHEKWFQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 6LK46DLK3C7A5HKHXBI4H6P2RF57HRAEONE744K62IIWIKFRFEJA) context deadline exceeded ======LOGS====== Dec 28 22:29:03.234193464 (559h0m3.105276499s) [upload=490074]: Starting upload Dec 28 22:29:03.234253228 (559h0m3.105336274s) [upload=490074]: Done starting upload Dec 28 22:29:03.234343239 (559h0m3.105426284s) [upload=490074]: Uploading segments... Dec 28 22:29:03.234476433 (559h0m3.105559468s) [upload=490074]: Got next segment. Inline: false Dec 28 22:29:03.234539113 (559h0m3.105622148s) [upload=490074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:29:03.234545825 (559h0m3.10562887s) [upload=490074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:29:03.480992019 (559h0m3.352075054s) [seg_pos={0 0}, upload=490074]: Begin upload segment... Dec 28 22:29:03.481045751 (559h0m3.352128786s) [seg_pos={0 0}, upload=490074]: Done begin upload segment. Dec 28 22:29:03.481090216 (559h0m3.352173251s) [noise=true , address=145.79.227.6:4259 , piece_id=6TDR5BWNEXSSPZRWMU2KNALEQGQTHCNQB5CWRX3XQHN2VTCE2EOA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.481132456 (559h0m3.352215491s) [noise=true , address=145.79.227.48:4302 , piece_id=OYVDCHEUYD37EK5MN3WAWH3LGCNL5T454JU5EDS4LHYESGAEWNWQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.481935585 (559h0m3.35301862s) [noise=true , address=45.91.116.4:4940 , piece_id=J226BBVOC5OOL7GFBDQYKPMAXYLR54ZNI6L2JPFGDVN6TLLX4VAA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.482331749 (559h0m3.353414794s) [noise=true , address=45.91.116.5:4297 , piece_id=HAI53YOMCKCJG5ZYNK3N4YAQSKLKB6KUAAKV34LAZKLDV2WJ4LGQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.482693668 (559h0m3.353776703s) [noise=true , address=145.79.227.50:4373 , piece_id=JQJXC3LNTT6XNQ3EQ5LBJQ2YP2LHPJNFQOHZH55IN6AXWDIBLXPQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.482820961 (559h0m3.353904006s) [noise=true , address=145.79.227.46:4602 , piece_id=7IHLOAS4LJUBJAV6CTXOLY5EVKDUU2KBI3I5UIVXMFMTSUHEBGYQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.483191526 (559h0m3.354274571s) [noise=true , address=145.79.227.7:4295 , piece_id=OZHNR5LJIEGABZSPP24BLKN2M5X2XUOODSMBBSF2GBA7YO6FFWEQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.483565929 (559h0m3.354648974s) [noise=true , address=145.79.227.47:4266 , piece_id=J6WFMSNTAC6REAJKINWXPIFPJMEKAG424GGK6CEORFU6TSFRJ5SA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.484064268 (559h0m3.355147313s) [noise=true , address=145.79.227.32:4834 , piece_id=R4PJDFDPLIOLMRMY4PMBJBGYNVCALBWQ6J75CS2ZIT4FZO4WEAKQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.484374909 (559h0m3.355457954s) [noise=true , address=45.91.116.13:4154 , piece_id=7LQPQ4BBVDLU7R5E2CJ6OVWJNXP6OWMREW77JCTKETK6GRHEKWFQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.484732259 (559h0m3.355815294s) [noise=true , address=145.79.227.44:4195 , piece_id=RSES7RLUFO3TXCAAT4LY2TK7VSE2BMXLPRNIUMIJGBQIH7D7VJAQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.485040046 (559h0m3.356123091s) [noise=true , address=145.79.227.52:4446 , piece_id=VIYSUY5RWVU4RKSRHH6XJV6XQCNSCXETE62DQRGUEFPRAAVMKLJQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.486251753 (559h0m3.357334798s) [noise=true , address=77.90.31.10:4518 , piece_id=R4L7ZIUOTVNT7QHFSAIXWDVU2IYC3EW7HSRVMECFKNGQSOISLXMA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.486794666 (559h0m3.357877701s) [noise=true , address=45.91.116.11:4117 , piece_id=6LK46DLK3C7A5HKHXBI4H6P2RF57HRAEONE744K62IIWIKFRFEJA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.487810641 (559h0m3.358893676s) [noise=true , address=77.90.31.11:4556 , piece_id=HI7EPN5GKQBSWU4LNSUEY76LMXPDYLG2Z5SQRMCHCZ6BYHZS34ZQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.488808671 (559h0m3.359891706s) [noise=true , address=145.79.227.5:4221 , piece_id=ZNKFRVH4FKX2OHYY7TFNXY7BL3ZCHGLEI4H3CPNN4TAIYLYEDZEA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:05.215604138 (559h0m5.086687173s) [upload=490074]: Next returned nil segment Dec 28 22:29:05.215618124 (559h0m5.086701159s) [upload=490074]: Waiting for error group managing segments... Dec 28 22:29:05.499544927 (559h0m5.370627963s) [noise=true , address=145.79.227.50:4373 , piece_id=JQJXC3LNTT6XNQ3EQ5LBJQ2YP2LHPJNFQOHZH55IN6AXWDIBLXPQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.508202160 (559h0m5.379285195s) [noise=true , address=145.79.227.52:4446 , piece_id=VIYSUY5RWVU4RKSRHH6XJV6XQCNSCXETE62DQRGUEFPRAAVMKLJQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.512343334 (559h0m5.383426369s) [noise=true , address=145.79.227.46:4602 , piece_id=7IHLOAS4LJUBJAV6CTXOLY5EVKDUU2KBI3I5UIVXMFMTSUHEBGYQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.512693731 (559h0m5.383776776s) [noise=true , address=145.79.227.47:4266 , piece_id=J6WFMSNTAC6REAJKINWXPIFPJMEKAG424GGK6CEORFU6TSFRJ5SA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.520833699 (559h0m5.391916745s) [noise=true , address=145.79.227.6:4259 , piece_id=6TDR5BWNEXSSPZRWMU2KNALEQGQTHCNQB5CWRX3XQHN2VTCE2EOA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.528284415 (559h0m5.39936745s) [noise=true , address=145.79.227.5:4221 , piece_id=ZNKFRVH4FKX2OHYY7TFNXY7BL3ZCHGLEI4H3CPNN4TAIYLYEDZEA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.535748456 (559h0m5.406831491s) [noise=true , address=145.79.227.48:4302 , piece_id=OYVDCHEUYD37EK5MN3WAWH3LGCNL5T454JU5EDS4LHYESGAEWNWQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.536951637 (559h0m5.408034682s) [noise=true , address=145.79.227.7:4295 , piece_id=OZHNR5LJIEGABZSPP24BLKN2M5X2XUOODSMBBSF2GBA7YO6FFWEQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.542701594 (559h0m5.41378463s) [noise=true , address=145.79.227.32:4834 , piece_id=R4PJDFDPLIOLMRMY4PMBJBGYNVCALBWQ6J75CS2ZIT4FZO4WEAKQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.734860740 (559h0m5.605943775s) [noise=true , address=77.90.31.11:4556 , piece_id=HI7EPN5GKQBSWU4LNSUEY76LMXPDYLG2Z5SQRMCHCZ6BYHZS34ZQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.738874121 (559h0m5.609957156s) [noise=true , address=45.91.116.11:4117 , piece_id=6LK46DLK3C7A5HKHXBI4H6P2RF57HRAEONE744K62IIWIKFRFEJA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.739730501 (559h0m5.610813536s) [noise=true , address=45.91.116.4:4940 , piece_id=J226BBVOC5OOL7GFBDQYKPMAXYLR54ZNI6L2JPFGDVN6TLLX4VAA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.758700138 (559h0m5.629783183s) [noise=true , address=45.91.116.13:4154 , piece_id=7LQPQ4BBVDLU7R5E2CJ6OVWJNXP6OWMREW77JCTKETK6GRHEKWFQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.758718813 (559h0m5.629801858s) [seg_pos={0 0}, upload=490074]: Segment reached optimal threshold of 13 pieces. Dec 28 22:29:05.758875382 (559h0m5.629958427s) [noise=true , address=77.90.31.10:4518 , piece_id=R4L7ZIUOTVNT7QHFSAIXWDVU2IYC3EW7HSRVMECFKNGQSOISLXMA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 28 22:29:05.758882054 (559h0m5.629965099s) [noise=true , address=77.90.31.10:4518 , piece_id=R4L7ZIUOTVNT7QHFSAIXWDVU2IYC3EW7HSRVMECFKNGQSOISLXMA, seg_pos={0 0}, upload=490074]: Long tail cancellation due to optimal threshold Dec 28 22:29:05.759057108 (559h0m5.630140183s) [noise=true , address=45.91.116.5:4297 , piece_id=HAI53YOMCKCJG5ZYNK3N4YAQSKLKB6KUAAKV34LAZKLDV2WJ4LGQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 22:29:05.759067457 (559h0m5.630150502s) [noise=true , address=45.91.116.5:4297 , piece_id=HAI53YOMCKCJG5ZYNK3N4YAQSKLKB6KUAAKV34LAZKLDV2WJ4LGQ, seg_pos={0 0}, upload=490074]: Long tail cancellation due to optimal threshold Dec 28 22:29:05.759353942 (559h0m5.630436997s) [noise=true , address=145.79.227.44:4195 , piece_id=RSES7RLUFO3TXCAAT4LY2TK7VSE2BMXLPRNIUMIJGBQIH7D7VJAQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 22:29:05.759365245 (559h0m5.63044828s) [noise=true , address=145.79.227.44:4195 , piece_id=RSES7RLUFO3TXCAAT4LY2TK7VSE2BMXLPRNIUMIJGBQIH7D7VJAQ, seg_pos={0 0}, upload=490074]: Long tail cancellation due to optimal threshold Dec 28 22:29:05.759389921 (559h0m5.630472966s) [seg_pos={0 0}, upload=490074]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:29:05.759472529 (559h0m5.630555574s) [upload=490074]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:29:05.759479372 (559h0m5.630562407s) [upload=490074]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:29:06.046694816 (559h0m5.917777851s) [upload=490074]: Done uploading segments... Dec 28 22:29:06.046712911 (559h0m5.917795946s) [upload=490074]: Upload finished. err: <nil> Duration: 6.32030333s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X....................X...] duration: 20.453306592s trace: https://jaeger.dev.storj.io/trace/46050a20bb3702b9 FAIL (2025-12-28 22:32:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 7LQPQ4BBVDLU7R5E2CJ6OVWJNXP6OWMREW77JCTKETK6GRHEKWFQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 6LK46DLK3C7A5HKHXBI4H6P2RF57HRAEONE744K62IIWIKFRFEJA) context deadline exceeded ======LOGS====== Dec 28 22:29:03.234193464 (559h0m3.105276499s) [upload=490074]: Starting upload Dec 28 22:29:03.234253228 (559h0m3.105336274s) [upload=490074]: Done starting upload Dec 28 22:29:03.234343239 (559h0m3.105426284s) [upload=490074]: Uploading segments... Dec 28 22:29:03.234476433 (559h0m3.105559468s) [upload=490074]: Got next segment. Inline: false Dec 28 22:29:03.234539113 (559h0m3.105622148s) [upload=490074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:29:03.234545825 (559h0m3.10562887s) [upload=490074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:29:03.480992019 (559h0m3.352075054s) [seg_pos={0 0}, upload=490074]: Begin upload segment... Dec 28 22:29:03.481045751 (559h0m3.352128786s) [seg_pos={0 0}, upload=490074]: Done begin upload segment. Dec 28 22:29:03.481090216 (559h0m3.352173251s) [noise=true , address=145.79.227.6:4259 , piece_id=6TDR5BWNEXSSPZRWMU2KNALEQGQTHCNQB5CWRX3XQHN2VTCE2EOA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.481132456 (559h0m3.352215491s) [noise=true , address=145.79.227.48:4302 , piece_id=OYVDCHEUYD37EK5MN3WAWH3LGCNL5T454JU5EDS4LHYESGAEWNWQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.481935585 (559h0m3.35301862s) [noise=true , address=45.91.116.4:4940 , piece_id=J226BBVOC5OOL7GFBDQYKPMAXYLR54ZNI6L2JPFGDVN6TLLX4VAA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.482331749 (559h0m3.353414794s) [noise=true , address=45.91.116.5:4297 , piece_id=HAI53YOMCKCJG5ZYNK3N4YAQSKLKB6KUAAKV34LAZKLDV2WJ4LGQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.482693668 (559h0m3.353776703s) [noise=true , address=145.79.227.50:4373 , piece_id=JQJXC3LNTT6XNQ3EQ5LBJQ2YP2LHPJNFQOHZH55IN6AXWDIBLXPQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.482820961 (559h0m3.353904006s) [noise=true , address=145.79.227.46:4602 , piece_id=7IHLOAS4LJUBJAV6CTXOLY5EVKDUU2KBI3I5UIVXMFMTSUHEBGYQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.483191526 (559h0m3.354274571s) [noise=true , address=145.79.227.7:4295 , piece_id=OZHNR5LJIEGABZSPP24BLKN2M5X2XUOODSMBBSF2GBA7YO6FFWEQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.483565929 (559h0m3.354648974s) [noise=true , address=145.79.227.47:4266 , piece_id=J6WFMSNTAC6REAJKINWXPIFPJMEKAG424GGK6CEORFU6TSFRJ5SA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.484064268 (559h0m3.355147313s) [noise=true , address=145.79.227.32:4834 , piece_id=R4PJDFDPLIOLMRMY4PMBJBGYNVCALBWQ6J75CS2ZIT4FZO4WEAKQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.484374909 (559h0m3.355457954s) [noise=true , address=45.91.116.13:4154 , piece_id=7LQPQ4BBVDLU7R5E2CJ6OVWJNXP6OWMREW77JCTKETK6GRHEKWFQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.484732259 (559h0m3.355815294s) [noise=true , address=145.79.227.44:4195 , piece_id=RSES7RLUFO3TXCAAT4LY2TK7VSE2BMXLPRNIUMIJGBQIH7D7VJAQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.485040046 (559h0m3.356123091s) [noise=true , address=145.79.227.52:4446 , piece_id=VIYSUY5RWVU4RKSRHH6XJV6XQCNSCXETE62DQRGUEFPRAAVMKLJQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.486251753 (559h0m3.357334798s) [noise=true , address=77.90.31.10:4518 , piece_id=R4L7ZIUOTVNT7QHFSAIXWDVU2IYC3EW7HSRVMECFKNGQSOISLXMA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.486794666 (559h0m3.357877701s) [noise=true , address=45.91.116.11:4117 , piece_id=6LK46DLK3C7A5HKHXBI4H6P2RF57HRAEONE744K62IIWIKFRFEJA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.487810641 (559h0m3.358893676s) [noise=true , address=77.90.31.11:4556 , piece_id=HI7EPN5GKQBSWU4LNSUEY76LMXPDYLG2Z5SQRMCHCZ6BYHZS34ZQ, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:03.488808671 (559h0m3.359891706s) [noise=true , address=145.79.227.5:4221 , piece_id=ZNKFRVH4FKX2OHYY7TFNXY7BL3ZCHGLEI4H3CPNN4TAIYLYEDZEA, seg_pos={0 0}, upload=490074]: Uploading piece... Dec 28 22:29:05.215604138 (559h0m5.086687173s) [upload=490074]: Next returned nil segment Dec 28 22:29:05.215618124 (559h0m5.086701159s) [upload=490074]: Waiting for error group managing segments... Dec 28 22:29:05.499544927 (559h0m5.370627963s) [noise=true , address=145.79.227.50:4373 , piece_id=JQJXC3LNTT6XNQ3EQ5LBJQ2YP2LHPJNFQOHZH55IN6AXWDIBLXPQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.508202160 (559h0m5.379285195s) [noise=true , address=145.79.227.52:4446 , piece_id=VIYSUY5RWVU4RKSRHH6XJV6XQCNSCXETE62DQRGUEFPRAAVMKLJQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.512343334 (559h0m5.383426369s) [noise=true , address=145.79.227.46:4602 , piece_id=7IHLOAS4LJUBJAV6CTXOLY5EVKDUU2KBI3I5UIVXMFMTSUHEBGYQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.512693731 (559h0m5.383776776s) [noise=true , address=145.79.227.47:4266 , piece_id=J6WFMSNTAC6REAJKINWXPIFPJMEKAG424GGK6CEORFU6TSFRJ5SA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.520833699 (559h0m5.391916745s) [noise=true , address=145.79.227.6:4259 , piece_id=6TDR5BWNEXSSPZRWMU2KNALEQGQTHCNQB5CWRX3XQHN2VTCE2EOA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.528284415 (559h0m5.39936745s) [noise=true , address=145.79.227.5:4221 , piece_id=ZNKFRVH4FKX2OHYY7TFNXY7BL3ZCHGLEI4H3CPNN4TAIYLYEDZEA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.535748456 (559h0m5.406831491s) [noise=true , address=145.79.227.48:4302 , piece_id=OYVDCHEUYD37EK5MN3WAWH3LGCNL5T454JU5EDS4LHYESGAEWNWQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.536951637 (559h0m5.408034682s) [noise=true , address=145.79.227.7:4295 , piece_id=OZHNR5LJIEGABZSPP24BLKN2M5X2XUOODSMBBSF2GBA7YO6FFWEQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.542701594 (559h0m5.41378463s) [noise=true , address=145.79.227.32:4834 , piece_id=R4PJDFDPLIOLMRMY4PMBJBGYNVCALBWQ6J75CS2ZIT4FZO4WEAKQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.734860740 (559h0m5.605943775s) [noise=true , address=77.90.31.11:4556 , piece_id=HI7EPN5GKQBSWU4LNSUEY76LMXPDYLG2Z5SQRMCHCZ6BYHZS34ZQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.738874121 (559h0m5.609957156s) [noise=true , address=45.91.116.11:4117 , piece_id=6LK46DLK3C7A5HKHXBI4H6P2RF57HRAEONE744K62IIWIKFRFEJA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.739730501 (559h0m5.610813536s) [noise=true , address=45.91.116.4:4940 , piece_id=J226BBVOC5OOL7GFBDQYKPMAXYLR54ZNI6L2JPFGDVN6TLLX4VAA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.758700138 (559h0m5.629783183s) [noise=true , address=45.91.116.13:4154 , piece_id=7LQPQ4BBVDLU7R5E2CJ6OVWJNXP6OWMREW77JCTKETK6GRHEKWFQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: <nil> Dec 28 22:29:05.758718813 (559h0m5.629801858s) [seg_pos={0 0}, upload=490074]: Segment reached optimal threshold of 13 pieces. Dec 28 22:29:05.758875382 (559h0m5.629958427s) [noise=true , address=77.90.31.10:4518 , piece_id=R4L7ZIUOTVNT7QHFSAIXWDVU2IYC3EW7HSRVMECFKNGQSOISLXMA, seg_pos={0 0}, upload=490074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 28 22:29:05.758882054 (559h0m5.629965099s) [noise=true , address=77.90.31.10:4518 , piece_id=R4L7ZIUOTVNT7QHFSAIXWDVU2IYC3EW7HSRVMECFKNGQSOISLXMA, seg_pos={0 0}, upload=490074]: Long tail cancellation due to optimal threshold Dec 28 22:29:05.759057108 (559h0m5.630140183s) [noise=true , address=45.91.116.5:4297 , piece_id=HAI53YOMCKCJG5ZYNK3N4YAQSKLKB6KUAAKV34LAZKLDV2WJ4LGQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 22:29:05.759067457 (559h0m5.630150502s) [noise=true , address=45.91.116.5:4297 , piece_id=HAI53YOMCKCJG5ZYNK3N4YAQSKLKB6KUAAKV34LAZKLDV2WJ4LGQ, seg_pos={0 0}, upload=490074]: Long tail cancellation due to optimal threshold Dec 28 22:29:05.759353942 (559h0m5.630436997s) [noise=true , address=145.79.227.44:4195 , piece_id=RSES7RLUFO3TXCAAT4LY2TK7VSE2BMXLPRNIUMIJGBQIH7D7VJAQ, seg_pos={0 0}, upload=490074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 22:29:05.759365245 (559h0m5.63044828s) [noise=true , address=145.79.227.44:4195 , piece_id=RSES7RLUFO3TXCAAT4LY2TK7VSE2BMXLPRNIUMIJGBQIH7D7VJAQ, seg_pos={0 0}, upload=490074]: Long tail cancellation due to optimal threshold Dec 28 22:29:05.759389921 (559h0m5.630472966s) [seg_pos={0 0}, upload=490074]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:29:05.759472529 (559h0m5.630555574s) [upload=490074]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:29:05.759479372 (559h0m5.630562407s) [upload=490074]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:29:06.046694816 (559h0m5.917777851s) [upload=490074]: Done uploading segments... Dec 28 22:29:06.046712911 (559h0m5.917795946s) [upload=490074]: Upload finished. err: <nil> Duration: 20.453331339s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)