Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 35.83358148s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:35:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 35.83361331s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXX....X...X..........X......XX.X.............X...X........X] duration: 30.003475382s trace: https://jaeger.dev.storj.io/trace/5288be00ddc0bf62 FAIL (2025-12-30 00:35:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 93857274s) [upload=512907]: Uploading segments... Dec 30 00:34:51.222865723 (585h5m51.093948759s) [upload=512907]: Got next segment. Inline: false Dec 30 00:34:51.222917001 (585h5m51.094000036s) [upload=512907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:34:51.222924205 (585h5m51.09400725s) [upload=512907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:34:51.466103374 (585h5m51.337186409s) [seg_pos={0 0}, upload=512907]: Begin upload segment... Dec 30 00:34:51.466164921 (585h5m51.337247966s) [seg_pos={0 0}, upload=512907]: Done begin upload segment. Dec 30 00:34:51.466224284 (585h5m51.337307329s) [noise=true , address=145.79.227.19:4077 , piece_id=47LY2L64PJ3MKQ5CUALWKX52HXOMYDFECS4U5T2QJ4D7USJ75FYQ, seg_pos={0 0}, upload=512907]: Uploading piece... Dec 30 00:34:51.466231538 (585h5m51.337314573s) [noise=true , address=45.91.116.6:4973 , piece_id=WASWHF5FMUHOQF7NDTEMD2KGOO326BQBKASUMJQCCGATGFQKSICQ, seg_pos={0 0}, upload=512907]: Uploading piece... Dec 30 00:34:51.466896594 (585h5m51.337979629s) [noise=true , address=145.79.227.43:4159 , piece_id=RAXK3AIM7UNIYSAZMAVKOA5EJ7ZEFGGXJPAI3F2SPPA7EOSRID2A, seg_pos={0 0}, upload=512907]: Uploading piece... Dec 30 00:34:51.467252542 (585h5m51.338335577s) [noise=true , address=145.79.227.11:4732 , piece_id=SIKTPQDJ4I6SNJX3ZJJBYGUJKFZRG4X445YWHOBJPHMYJKI4T5LA, seg_pos={0 0}, upload=512907]: Uploading piece... Dec 30 00:34:51.467347984 (585h5m51.338431029s) [noise=true , address=145.79.227.51:4410 , piece_id=RFWBJJ2E2DDWK5UPKYM5LUEIZKOF7U7QFCHMS3RHOB57D3EUI6UA, seg_pos={0 0}, upload=512907]: Uploading piece... Dec 30 00:34:51.467713870 (585h5m51.338796915s) [noise=true , address=145.79.227.12:4767 , piece_id=NOYNBVA2IH57V2YD7ZYHR6TIOYNYPT5VM3A3SA3YLBHIQTZ56TXQ, seg_pos={0 0}, upload=512907]: Uploading piece... Dec 30 00:34:51.467718719 (585h5m51.338801764s) [noise=true , address=145.79.227.41:4087 , piece_id=WDXWYICZSPLQTCOXEDDCNAJCIXYZQN4G774TNNDFMUBER6FRQF3A, seg_pos={0 0}, upload=512907]: Uploading piece... Dec 30 00:34:51.468036284 (585h5m51.339119319s) [noise=true , address=145.79.227.18:4006 , piece_id=BQBDF3DYNTGTTBKHBY4RJTWUEKKMLZ7HD64SOIIP6WOS5UG7D2TA, seg_pos={0 0}, upload=512907]: Uploading piece... Dec 30 00:34:51.468079076 (585h5m51.339162121s) [noise=true , address=145.79.227.37:4566 , piece_id=JG67ES4LP66RGZXTGFOGUKXBR5MDJYADQRBL3ATQX7IIUXOWCDBQ, seg_pos={0 0}, upload=512907]: Uploading piece... Dec 30 00:34:51.468372113 (585h5m51.339455149s) [noise=true , address=145.79.227.26:4653 , piece_id=WHKUOCQ3KOSXEBJ7SF3RGCKGZTP65Z4LY67F7SDGYGX46335KPYQ, seg_pos={0 0}, upload=512907]: Uploading piece... Dec 30 00:34:51.468468297 (585h5m51.339551342s) [noise=true , address=45.91.116.13:4153 , piece_id=NPEHWPZBEDDFSLBKFAK4NMBCVRDE4XIS6P3TNA4HHNF6MC2TWF4Q, seg_pos={0 0}, upload=512907]: Uploading piece... Dec 30 00:34:51.468703554 (585h5m51.339786599s) [noise=true , address=45.91.116.8:4046 , piece_id=AIIJJWO5K3CTESR44RNCO7SI3MS2I25R7WJUVGXZWLNL5U3QRZJA, seg_pos={0 0}, upload=512907]: Uploading piece... Dec 30 00:34:51.468979770 (585h5m51.340062805s) [noise=true , address=145.79.227.50:4374 , piece_id=5HCHSOF3NHOUKOU55JU6IZC3LE3EOIOWNAVKODPLI5JZT3RW3Z3A, seg_pos={0 0}, upload=512907]: Uploading piece... Dec 30 00:34:51.469024686 (585h5m51.340107731s) [noise=true , address=77.90.31.10:4518 , piece_id=QSPWDHWABNLI3LKQIMERMV4L45AWBPAV3PASB7TRWSUNHCRKW3UQ, seg_pos={0 0}, upload=512907]: Uploading piece... Dec 30 00:34:51.469358391 (585h5m51.340441436s) [noise=true , address=145.79.227.14:4839 , piece_id=NJQPTRPPLUIDZDMQMYUIODNJQVWFYR3S345RKMHCDYFU5IGOYXMA, seg_pos={0 0}, upload=512907]: Uploading piece... Dec 30 00:34:51.469412314 (585h5m51.34049543s) [noise=true , address=145.79.227.10:4402 , piece_id=II3PA2RRFUBF5T6IATBXPKHNJPMNQATW3UEQAMOTG423GA7G2J7Q, seg_pos={0 0}, upload=512907]: Uploading piece... Dec 30 00:34:53.111139762 (585h5m52.982222807s) [upload=512907]: Next returned nil segment Dec 30 00:34:53.111153007 (585h5m52.982236043s) [upload=512907]: Waiting for error group managing segments... Dec 30 00:34:53.442645962 (585h5m53.313729007s) [noise=true , address=145.79.227.18:4006 , piece_id=BQBDF3DYNTGTTBKHBY4RJTWUEKKMLZ7HD64SOIIP6WOS5UG7D2TA, seg_pos={0 0}, upload=512907]: Done uploading piece. err: <nil> Dec 30 00:34:53.448789770 (585h5m53.319872805s) [noise=true , address=145.79.227.14:4839 , piece_id=NJQPTRPPLUIDZDMQMYUIODNJQVWFYR3S345RKMHCDYFU5IGOYXMA, seg_pos={0 0}, upload=512907]: Done uploading piece. err: <nil> Dec 30 00:34:53.448843522 (585h5m53.319926557s) [noise=true , address=145.79.227.10:4402 , piece_id=II3PA2RRFUBF5T6IATBXPKHNJPMNQATW3UEQAMOTG423GA7G2J7Q, seg_pos={0 0}, upload=512907]: Done uploading piece. err: <nil> Dec 30 00:34:53.451482457 (585h5m53.322565502s) [noise=true , address=145.79.227.12:4767 , piece_id=NOYNBVA2IH57V2YD7ZYHR6TIOYNYPT5VM3A3SA3YLBHIQTZ56TXQ, seg_pos={0 0}, upload=512907]: Done uploading piece. err: <nil> Dec 30 00:34:53.453698406 (585h5m53.324781441s) [noise=true , address=145.79.227.26:4653 , piece_id=WHKUOCQ3KOSXEBJ7SF3RGCKGZTP65Z4LY67F7SDGYGX46335KPYQ, seg_pos={0 0}, upload=512907]: Done uploading piece. err: <nil> Dec 30 00:34:53.458380923 (585h5m53.329463968s) [noise=true , address=145.79.227.11:4732 , piece_id=SIKTPQDJ4I6SNJX3ZJJBYGUJKFZRG4X445YWHOBJPHMYJKI4T5LA, seg_pos={0 0}, upload=512907]: Done uploading piece. err: <nil> Dec 30 00:34:53.463067616 (585h5m53.334150661s) [noise=true , address=145.79.227.51:4410 , piece_id=RFWBJJ2E2DDWK5UPKYM5LUEIZKOF7U7QFCHMS3RHOB57D3EUI6UA, seg_pos={0 0}, upload=512907]: Done uploading piece. err: <nil> Dec 30 00:34:53.463219225 (585h5m53.33430226s) [noise=true , address=145.79.227.37:4566 , piece_id=JG67ES4LP66RGZXTGFOGUKXBR5MDJYADQRBL3ATQX7IIUXOWCDBQ, seg_pos={0 0}, upload=512907]: Done uploading piece. err: <nil> Dec 30 00:34:53.465963571 (585h5m53.337046616s) [noise=true , address=145.79.227.50:4374 , piece_id=5HCHSOF3NHOUKOU55JU6IZC3LE3EOIOWNAVKODPLI5JZT3RW3Z3A, seg_pos={0 0}, upload=512907]: Done uploading piece. err: <nil> Dec 30 00:34:53.468029133 (585h5m53.339112178s) [noise=true , address=145.79.227.41:4087 , piece_id=WDXWYICZSPLQTCOXEDDCNAJCIXYZQN4G774TNNDFMUBER6FRQF3A, seg_pos={0 0}, upload=512907]: Done uploading piece. err: <nil> Dec 30 00:34:53.614970810 (585h5m53.486053845s) [noise=true , address=77.90.31.10:4518 , piece_id=QSPWDHWABNLI3LKQIMERMV4L45AWBPAV3PASB7TRWSUNHCRKW3UQ, seg_pos={0 0}, upload=512907]: Done uploading piece. err: <nil> Dec 30 00:34:53.702043692 (585h5m53.573126737s) [noise=true , address=45.91.116.6:4973 , piece_id=WASWHF5FMUHOQF7NDTEMD2KGOO326BQBKASUMJQCCGATGFQKSICQ, seg_pos={0 0}, upload=512907]: Done uploading piece. err: <nil> Dec 30 00:34:53.799017347 (585h5m53.670100382s) [noise=true , address=45.91.116.8:4046 , piece_id=AIIJJWO5K3CTESR44RNCO7SI3MS2I25R7WJUVGXZWLNL5U3QRZJA, seg_pos={0 0}, upload=512907]: Done uploading piece. err: <nil> Dec 30 00:34:53.799039999 (585h5m53.670123044s) [seg_pos={0 0}, upload=512907]: Segment reached optimal threshold of 13 pieces. Dec 30 00:34:53.799293151 (585h5m53.670376186s) [noise=true , address=145.79.227.43:4159 , piece_id=RAXK3AIM7UNIYSAZMAVKOA5EJ7ZEFGGXJPAI3F2SPPA7EOSRID2A, seg_pos={0 0}, upload=512907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 30 00:34:53.799302700 (585h5m53.670385735s) [noise=true , address=145.79.227.43:4159 , piece_id=RAXK3AIM7UNIYSAZMAVKOA5EJ7ZEFGGXJPAI3F2SPPA7EOSRID2A, seg_pos={0 0}, upload=512907]: Long tail cancellation due to optimal threshold Dec 30 00:34:53.799513051 (585h5m53.670596086s) [noise=true , address=145.79.227.19:4077 , piece_id=47LY2L64PJ3MKQ5CUALWKX52HXOMYDFECS4U5T2QJ4D7USJ75FYQ, seg_pos={0 0}, upload=512907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:35754->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:35754->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35754->145.79.227.19:4077: use of closed network connection Dec 30 00:34:53.799519433 (585h5m53.670602468s) [noise=true , address=145.79.227.19:4077 , piece_id=47LY2L64PJ3MKQ5CUALWKX52HXOMYDFECS4U5T2QJ4D7USJ75FYQ, seg_pos={0 0}, upload=512907]: Long tail cancellation due to optimal threshold Dec 30 00:34:53.799530463 (585h5m53.670613518s) [noise=true , address=45.91.116.13:4153 , piece_id=NPEHWPZBEDDFSLBKFAK4NMBCVRDE4XIS6P3TNA4HHNF6MC2TWF4Q, seg_pos={0 0}, upload=512907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:42496->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:42496->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42496->45.91.116.13:4153: use of closed network connection Dec 30 00:34:53.799573184 (585h5m53.670656229s) [noise=true , address=45.91.116.13:4153 , piece_id=NPEHWPZBEDDFSLBKFAK4NMBCVRDE4XIS6P3TNA4HHNF6MC2TWF4Q, seg_pos={0 0}, upload=512907]: Long tail cancellation due to optimal threshold Dec 30 00:34:53.799681892 (585h5m53.670764927s) [seg_pos={0 0}, upload=512907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:34:53.799757747 (585h5m53.670840792s) [upload=512907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:34:53.799766363 (585h5m53.670849428s) [upload=512907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:34:54.091974557 (585h5m53.963057592s) [upload=512907]: Done uploading segments... Dec 30 00:34:54.091992591 (585h5m53.963075626s) [upload=512907]: Upload finished. err: <nil> Duration: 30.003487816s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.558755074s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:34:24 UTC) orbiter warning: context deadline exceeded Duration: 3.558829536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 15.409087505s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:34:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 15.409114396s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.XXX....X...X..........X......XX.X.............X...X........] duration: 6.314589169s trace: https://jaeger.dev.storj.io/trace/4576e0b260f1cccc FAIL (2025-12-30 00:34:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA) context deadline exceeded ======LOGS====== egments... Dec 30 00:28:21.635917127 (584h59m21.507000173s) [upload=512805]: Got next segment. Inline: false Dec 30 00:28:21.635940682 (584h59m21.507023717s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:28:21.635944719 (584h59m21.507027754s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:28:21.881257599 (584h59m21.752340644s) [seg_pos={0 0}, upload=512805]: Begin upload segment... Dec 30 00:28:21.881299850 (584h59m21.752382885s) [seg_pos={0 0}, upload=512805]: Done begin upload segment. Dec 30 00:28:21.881324306 (584h59m21.752407341s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.881389700 (584h59m21.752472745s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.882087338 (584h59m21.753170374s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883576975 (584h59m21.75466001s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883586743 (584h59m21.754669779s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884026291 (584h59m21.755109336s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884040708 (584h59m21.755123743s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884417556 (584h59m21.755500601s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884441141 (584h59m21.755524176s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884776419 (584h59m21.755859464s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884844208 (584h59m21.755927243s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885201499 (584h59m21.756284534s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885560672 (584h59m21.756643708s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885971014 (584h59m21.757054069s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.886602937 (584h59m21.757685972s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.887486650 (584h59m21.758569695s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:23.486845051 (584h59m23.357928086s) [upload=512805]: Next returned nil segment Dec 30 00:28:23.486866302 (584h59m23.357949337s) [upload=512805]: Waiting for error group managing segments... Dec 30 00:28:23.868031110 (584h59m23.739114165s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.871946364 (584h59m23.743029399s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874212258 (584h59m23.745295303s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874377704 (584h59m23.745460739s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874851677 (584h59m23.745934712s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.878945862 (584h59m23.750028897s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.881991060 (584h59m23.753074105s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.887715971 (584h59m23.758799006s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.054678917 (584h59m23.925761952s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.109927864 (584h59m23.981010899s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.139250515 (584h59m24.01033355s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.341783439 (584h59m24.212866484s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436862164 (584h59m24.307945199s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436881480 (584h59m24.307964525s) [seg_pos={0 0}, upload=512805]: Segment reached optimal threshold of 13 pieces. Dec 30 00:28:24.437052376 (584h59m24.308135421s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 00:28:24.437058267 (584h59m24.308141312s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437174368 (584h59m24.308257403s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection Dec 30 00:28:24.437182995 (584h59m24.30826604s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437218943 (584h59m24.308301978s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection Dec 30 00:28:24.437224754 (584h59m24.308307789s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437238090 (584h59m24.308321125s) [seg_pos={0 0}, upload=512805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:28:24.437298264 (584h59m24.308381309s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:28:24.437302632 (584h59m24.308385677s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:28:24.726409577 (584h59m24.597492613s) [upload=512805]: Done uploading segments... Dec 30 00:28:24.726519537 (584h59m24.597602572s) [upload=512805]: Upload finished. err: <nil> Duration: 6.31462629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 15.596559849s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:33:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 15.59658694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..XXX....X...X..........X......XX.X.............X...X.......] duration: 5.828733202s trace: https://jaeger.dev.storj.io/trace/4576e0b260f1cccc FAIL (2025-12-30 00:33:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA) context deadline exceeded ======LOGS====== egments... Dec 30 00:28:21.635917127 (584h59m21.507000173s) [upload=512805]: Got next segment. Inline: false Dec 30 00:28:21.635940682 (584h59m21.507023717s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:28:21.635944719 (584h59m21.507027754s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:28:21.881257599 (584h59m21.752340644s) [seg_pos={0 0}, upload=512805]: Begin upload segment... Dec 30 00:28:21.881299850 (584h59m21.752382885s) [seg_pos={0 0}, upload=512805]: Done begin upload segment. Dec 30 00:28:21.881324306 (584h59m21.752407341s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.881389700 (584h59m21.752472745s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.882087338 (584h59m21.753170374s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883576975 (584h59m21.75466001s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883586743 (584h59m21.754669779s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884026291 (584h59m21.755109336s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884040708 (584h59m21.755123743s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884417556 (584h59m21.755500601s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884441141 (584h59m21.755524176s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884776419 (584h59m21.755859464s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884844208 (584h59m21.755927243s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885201499 (584h59m21.756284534s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885560672 (584h59m21.756643708s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885971014 (584h59m21.757054069s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.886602937 (584h59m21.757685972s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.887486650 (584h59m21.758569695s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:23.486845051 (584h59m23.357928086s) [upload=512805]: Next returned nil segment Dec 30 00:28:23.486866302 (584h59m23.357949337s) [upload=512805]: Waiting for error group managing segments... Dec 30 00:28:23.868031110 (584h59m23.739114165s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.871946364 (584h59m23.743029399s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874212258 (584h59m23.745295303s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874377704 (584h59m23.745460739s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874851677 (584h59m23.745934712s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.878945862 (584h59m23.750028897s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.881991060 (584h59m23.753074105s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.887715971 (584h59m23.758799006s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.054678917 (584h59m23.925761952s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.109927864 (584h59m23.981010899s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.139250515 (584h59m24.01033355s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.341783439 (584h59m24.212866484s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436862164 (584h59m24.307945199s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436881480 (584h59m24.307964525s) [seg_pos={0 0}, upload=512805]: Segment reached optimal threshold of 13 pieces. Dec 30 00:28:24.437052376 (584h59m24.308135421s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 00:28:24.437058267 (584h59m24.308141312s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437174368 (584h59m24.308257403s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection Dec 30 00:28:24.437182995 (584h59m24.30826604s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437218943 (584h59m24.308301978s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection Dec 30 00:28:24.437224754 (584h59m24.308307789s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437238090 (584h59m24.308321125s) [seg_pos={0 0}, upload=512805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:28:24.437298264 (584h59m24.308381309s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:28:24.437302632 (584h59m24.308385677s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:28:24.726409577 (584h59m24.597492613s) [upload=512805]: Done uploading segments... Dec 30 00:28:24.726519537 (584h59m24.597602572s) [upload=512805]: Upload finished. err: <nil> Duration: 5.828756547s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.561239826s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:33:18 UTC) orbiter warning: context deadline exceeded Duration: 3.56130515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 18.991043141s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:33:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 18.991103597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 15.485991403s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:32:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 15.486022452s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...XXX....X...X..........X......XX.X.............X...X......] duration: 14.564711566s trace: https://jaeger.dev.storj.io/trace/4576e0b260f1cccc FAIL (2025-12-30 00:32:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA) context deadline exceeded ======LOGS====== egments... Dec 30 00:28:21.635917127 (584h59m21.507000173s) [upload=512805]: Got next segment. Inline: false Dec 30 00:28:21.635940682 (584h59m21.507023717s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:28:21.635944719 (584h59m21.507027754s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:28:21.881257599 (584h59m21.752340644s) [seg_pos={0 0}, upload=512805]: Begin upload segment... Dec 30 00:28:21.881299850 (584h59m21.752382885s) [seg_pos={0 0}, upload=512805]: Done begin upload segment. Dec 30 00:28:21.881324306 (584h59m21.752407341s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.881389700 (584h59m21.752472745s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.882087338 (584h59m21.753170374s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883576975 (584h59m21.75466001s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883586743 (584h59m21.754669779s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884026291 (584h59m21.755109336s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884040708 (584h59m21.755123743s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884417556 (584h59m21.755500601s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884441141 (584h59m21.755524176s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884776419 (584h59m21.755859464s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884844208 (584h59m21.755927243s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885201499 (584h59m21.756284534s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885560672 (584h59m21.756643708s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885971014 (584h59m21.757054069s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.886602937 (584h59m21.757685972s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.887486650 (584h59m21.758569695s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:23.486845051 (584h59m23.357928086s) [upload=512805]: Next returned nil segment Dec 30 00:28:23.486866302 (584h59m23.357949337s) [upload=512805]: Waiting for error group managing segments... Dec 30 00:28:23.868031110 (584h59m23.739114165s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.871946364 (584h59m23.743029399s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874212258 (584h59m23.745295303s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874377704 (584h59m23.745460739s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874851677 (584h59m23.745934712s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.878945862 (584h59m23.750028897s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.881991060 (584h59m23.753074105s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.887715971 (584h59m23.758799006s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.054678917 (584h59m23.925761952s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.109927864 (584h59m23.981010899s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.139250515 (584h59m24.01033355s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.341783439 (584h59m24.212866484s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436862164 (584h59m24.307945199s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436881480 (584h59m24.307964525s) [seg_pos={0 0}, upload=512805]: Segment reached optimal threshold of 13 pieces. Dec 30 00:28:24.437052376 (584h59m24.308135421s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 00:28:24.437058267 (584h59m24.308141312s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437174368 (584h59m24.308257403s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection Dec 30 00:28:24.437182995 (584h59m24.30826604s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437218943 (584h59m24.308301978s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection Dec 30 00:28:24.437224754 (584h59m24.308307789s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437238090 (584h59m24.308321125s) [seg_pos={0 0}, upload=512805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:28:24.437298264 (584h59m24.308381309s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:28:24.437302632 (584h59m24.308385677s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:28:24.726409577 (584h59m24.597492613s) [upload=512805]: Done uploading segments... Dec 30 00:28:24.726519537 (584h59m24.597602572s) [upload=512805]: Upload finished. err: <nil> Duration: 14.564736573s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.540238519s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:32:10 UTC) orbiter warning: context deadline exceeded Duration: 3.540262934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....XXX....X...X..........X......XX.X.............X...X.....] duration: 6.093568871s trace: https://jaeger.dev.storj.io/trace/4576e0b260f1cccc FAIL (2025-12-30 00:31:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA) context deadline exceeded ======LOGS====== egments... Dec 30 00:28:21.635917127 (584h59m21.507000173s) [upload=512805]: Got next segment. Inline: false Dec 30 00:28:21.635940682 (584h59m21.507023717s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:28:21.635944719 (584h59m21.507027754s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:28:21.881257599 (584h59m21.752340644s) [seg_pos={0 0}, upload=512805]: Begin upload segment... Dec 30 00:28:21.881299850 (584h59m21.752382885s) [seg_pos={0 0}, upload=512805]: Done begin upload segment. Dec 30 00:28:21.881324306 (584h59m21.752407341s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.881389700 (584h59m21.752472745s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.882087338 (584h59m21.753170374s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883576975 (584h59m21.75466001s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883586743 (584h59m21.754669779s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884026291 (584h59m21.755109336s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884040708 (584h59m21.755123743s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884417556 (584h59m21.755500601s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884441141 (584h59m21.755524176s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884776419 (584h59m21.755859464s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884844208 (584h59m21.755927243s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885201499 (584h59m21.756284534s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885560672 (584h59m21.756643708s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885971014 (584h59m21.757054069s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.886602937 (584h59m21.757685972s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.887486650 (584h59m21.758569695s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:23.486845051 (584h59m23.357928086s) [upload=512805]: Next returned nil segment Dec 30 00:28:23.486866302 (584h59m23.357949337s) [upload=512805]: Waiting for error group managing segments... Dec 30 00:28:23.868031110 (584h59m23.739114165s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.871946364 (584h59m23.743029399s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874212258 (584h59m23.745295303s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874377704 (584h59m23.745460739s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874851677 (584h59m23.745934712s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.878945862 (584h59m23.750028897s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.881991060 (584h59m23.753074105s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.887715971 (584h59m23.758799006s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.054678917 (584h59m23.925761952s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.109927864 (584h59m23.981010899s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.139250515 (584h59m24.01033355s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.341783439 (584h59m24.212866484s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436862164 (584h59m24.307945199s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436881480 (584h59m24.307964525s) [seg_pos={0 0}, upload=512805]: Segment reached optimal threshold of 13 pieces. Dec 30 00:28:24.437052376 (584h59m24.308135421s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 00:28:24.437058267 (584h59m24.308141312s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437174368 (584h59m24.308257403s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection Dec 30 00:28:24.437182995 (584h59m24.30826604s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437218943 (584h59m24.308301978s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection Dec 30 00:28:24.437224754 (584h59m24.308307789s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437238090 (584h59m24.308321125s) [seg_pos={0 0}, upload=512805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:28:24.437298264 (584h59m24.308381309s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:28:24.437302632 (584h59m24.308385677s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:28:24.726409577 (584h59m24.597492613s) [upload=512805]: Done uploading segments... Dec 30 00:28:24.726519537 (584h59m24.597602572s) [upload=512805]: Upload finished. err: <nil> Duration: 6.093597566s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.548516832s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:31:38 UTC) orbiter warning: context deadline exceeded Duration: 3.548567789s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 15.445257705s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:31:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 15.445278625s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 32.73407584s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:31:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 32.734090498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 29.91797827s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:30:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 29.918008027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....XXX....X...X..........X......XX.X.............X...X....] duration: 5.80501427s trace: https://jaeger.dev.storj.io/trace/4576e0b260f1cccc FAIL (2025-12-30 00:30:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA) context deadline exceeded ======LOGS====== egments... Dec 30 00:28:21.635917127 (584h59m21.507000173s) [upload=512805]: Got next segment. Inline: false Dec 30 00:28:21.635940682 (584h59m21.507023717s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:28:21.635944719 (584h59m21.507027754s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:28:21.881257599 (584h59m21.752340644s) [seg_pos={0 0}, upload=512805]: Begin upload segment... Dec 30 00:28:21.881299850 (584h59m21.752382885s) [seg_pos={0 0}, upload=512805]: Done begin upload segment. Dec 30 00:28:21.881324306 (584h59m21.752407341s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.881389700 (584h59m21.752472745s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.882087338 (584h59m21.753170374s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883576975 (584h59m21.75466001s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883586743 (584h59m21.754669779s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884026291 (584h59m21.755109336s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884040708 (584h59m21.755123743s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884417556 (584h59m21.755500601s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884441141 (584h59m21.755524176s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884776419 (584h59m21.755859464s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884844208 (584h59m21.755927243s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885201499 (584h59m21.756284534s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885560672 (584h59m21.756643708s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885971014 (584h59m21.757054069s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.886602937 (584h59m21.757685972s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.887486650 (584h59m21.758569695s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:23.486845051 (584h59m23.357928086s) [upload=512805]: Next returned nil segment Dec 30 00:28:23.486866302 (584h59m23.357949337s) [upload=512805]: Waiting for error group managing segments... Dec 30 00:28:23.868031110 (584h59m23.739114165s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.871946364 (584h59m23.743029399s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874212258 (584h59m23.745295303s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874377704 (584h59m23.745460739s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874851677 (584h59m23.745934712s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.878945862 (584h59m23.750028897s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.881991060 (584h59m23.753074105s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.887715971 (584h59m23.758799006s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.054678917 (584h59m23.925761952s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.109927864 (584h59m23.981010899s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.139250515 (584h59m24.01033355s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.341783439 (584h59m24.212866484s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436862164 (584h59m24.307945199s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436881480 (584h59m24.307964525s) [seg_pos={0 0}, upload=512805]: Segment reached optimal threshold of 13 pieces. Dec 30 00:28:24.437052376 (584h59m24.308135421s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 00:28:24.437058267 (584h59m24.308141312s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437174368 (584h59m24.308257403s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection Dec 30 00:28:24.437182995 (584h59m24.30826604s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437218943 (584h59m24.308301978s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection Dec 30 00:28:24.437224754 (584h59m24.308307789s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437238090 (584h59m24.308321125s) [seg_pos={0 0}, upload=512805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:28:24.437298264 (584h59m24.308381309s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:28:24.437302632 (584h59m24.308385677s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:28:24.726409577 (584h59m24.597492613s) [upload=512805]: Done uploading segments... Dec 30 00:28:24.726519537 (584h59m24.597602572s) [upload=512805]: Upload finished. err: <nil> Duration: 5.805038767s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.459550853s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:30:38 UTC) orbiter warning: context deadline exceeded Duration: 3.459578687s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......XXX....X...X..........X......XX.X.............X...X...] duration: 22.422380857s trace: https://jaeger.dev.storj.io/trace/4576e0b260f1cccc FAIL (2025-12-30 00:30:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA) context deadline exceeded ======LOGS====== egments... Dec 30 00:28:21.635917127 (584h59m21.507000173s) [upload=512805]: Got next segment. Inline: false Dec 30 00:28:21.635940682 (584h59m21.507023717s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:28:21.635944719 (584h59m21.507027754s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:28:21.881257599 (584h59m21.752340644s) [seg_pos={0 0}, upload=512805]: Begin upload segment... Dec 30 00:28:21.881299850 (584h59m21.752382885s) [seg_pos={0 0}, upload=512805]: Done begin upload segment. Dec 30 00:28:21.881324306 (584h59m21.752407341s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.881389700 (584h59m21.752472745s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.882087338 (584h59m21.753170374s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883576975 (584h59m21.75466001s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883586743 (584h59m21.754669779s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884026291 (584h59m21.755109336s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884040708 (584h59m21.755123743s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884417556 (584h59m21.755500601s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884441141 (584h59m21.755524176s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884776419 (584h59m21.755859464s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884844208 (584h59m21.755927243s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885201499 (584h59m21.756284534s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885560672 (584h59m21.756643708s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885971014 (584h59m21.757054069s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.886602937 (584h59m21.757685972s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.887486650 (584h59m21.758569695s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:23.486845051 (584h59m23.357928086s) [upload=512805]: Next returned nil segment Dec 30 00:28:23.486866302 (584h59m23.357949337s) [upload=512805]: Waiting for error group managing segments... Dec 30 00:28:23.868031110 (584h59m23.739114165s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.871946364 (584h59m23.743029399s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874212258 (584h59m23.745295303s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874377704 (584h59m23.745460739s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874851677 (584h59m23.745934712s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.878945862 (584h59m23.750028897s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.881991060 (584h59m23.753074105s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.887715971 (584h59m23.758799006s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.054678917 (584h59m23.925761952s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.109927864 (584h59m23.981010899s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.139250515 (584h59m24.01033355s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.341783439 (584h59m24.212866484s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436862164 (584h59m24.307945199s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436881480 (584h59m24.307964525s) [seg_pos={0 0}, upload=512805]: Segment reached optimal threshold of 13 pieces. Dec 30 00:28:24.437052376 (584h59m24.308135421s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 00:28:24.437058267 (584h59m24.308141312s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437174368 (584h59m24.308257403s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection Dec 30 00:28:24.437182995 (584h59m24.30826604s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437218943 (584h59m24.308301978s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection Dec 30 00:28:24.437224754 (584h59m24.308307789s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437238090 (584h59m24.308321125s) [seg_pos={0 0}, upload=512805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:28:24.437298264 (584h59m24.308381309s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:28:24.437302632 (584h59m24.308385677s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:28:24.726409577 (584h59m24.597492613s) [upload=512805]: Done uploading segments... Dec 30 00:28:24.726519537 (584h59m24.597602572s) [upload=512805]: Upload finished. err: <nil> Duration: 22.422417697s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.505290344s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:30:00 UTC) orbiter warning: context deadline exceeded Duration: 3.505317516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 25.471817276s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:29:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 25.471834438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......XXX....X...X..........X......XX.X.............X...X..] duration: 6.090875989s trace: https://jaeger.dev.storj.io/trace/4576e0b260f1cccc FAIL (2025-12-30 00:29:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA) context deadline exceeded ======LOGS====== egments... Dec 30 00:28:21.635917127 (584h59m21.507000173s) [upload=512805]: Got next segment. Inline: false Dec 30 00:28:21.635940682 (584h59m21.507023717s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:28:21.635944719 (584h59m21.507027754s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:28:21.881257599 (584h59m21.752340644s) [seg_pos={0 0}, upload=512805]: Begin upload segment... Dec 30 00:28:21.881299850 (584h59m21.752382885s) [seg_pos={0 0}, upload=512805]: Done begin upload segment. Dec 30 00:28:21.881324306 (584h59m21.752407341s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.881389700 (584h59m21.752472745s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.882087338 (584h59m21.753170374s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883576975 (584h59m21.75466001s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883586743 (584h59m21.754669779s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884026291 (584h59m21.755109336s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884040708 (584h59m21.755123743s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884417556 (584h59m21.755500601s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884441141 (584h59m21.755524176s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884776419 (584h59m21.755859464s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884844208 (584h59m21.755927243s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885201499 (584h59m21.756284534s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885560672 (584h59m21.756643708s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885971014 (584h59m21.757054069s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.886602937 (584h59m21.757685972s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.887486650 (584h59m21.758569695s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:23.486845051 (584h59m23.357928086s) [upload=512805]: Next returned nil segment Dec 30 00:28:23.486866302 (584h59m23.357949337s) [upload=512805]: Waiting for error group managing segments... Dec 30 00:28:23.868031110 (584h59m23.739114165s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.871946364 (584h59m23.743029399s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874212258 (584h59m23.745295303s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874377704 (584h59m23.745460739s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874851677 (584h59m23.745934712s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.878945862 (584h59m23.750028897s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.881991060 (584h59m23.753074105s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.887715971 (584h59m23.758799006s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.054678917 (584h59m23.925761952s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.109927864 (584h59m23.981010899s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.139250515 (584h59m24.01033355s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.341783439 (584h59m24.212866484s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436862164 (584h59m24.307945199s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436881480 (584h59m24.307964525s) [seg_pos={0 0}, upload=512805]: Segment reached optimal threshold of 13 pieces. Dec 30 00:28:24.437052376 (584h59m24.308135421s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 00:28:24.437058267 (584h59m24.308141312s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437174368 (584h59m24.308257403s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection Dec 30 00:28:24.437182995 (584h59m24.30826604s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437218943 (584h59m24.308301978s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection Dec 30 00:28:24.437224754 (584h59m24.308307789s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437238090 (584h59m24.308321125s) [seg_pos={0 0}, upload=512805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:28:24.437298264 (584h59m24.308381309s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:28:24.437302632 (584h59m24.308385677s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:28:24.726409577 (584h59m24.597492613s) [upload=512805]: Done uploading segments... Dec 30 00:28:24.726519537 (584h59m24.597602572s) [upload=512805]: Upload finished. err: <nil> Duration: 6.090906797s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 25.858085621s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:29:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 25.858116629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [........XXX....X...X..........X......XX.X.............X...X.] duration: 7.683676993s trace: https://jaeger.dev.storj.io/trace/4576e0b260f1cccc FAIL (2025-12-30 00:29:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA) context deadline exceeded ======LOGS====== egments... Dec 30 00:28:21.635917127 (584h59m21.507000173s) [upload=512805]: Got next segment. Inline: false Dec 30 00:28:21.635940682 (584h59m21.507023717s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:28:21.635944719 (584h59m21.507027754s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:28:21.881257599 (584h59m21.752340644s) [seg_pos={0 0}, upload=512805]: Begin upload segment... Dec 30 00:28:21.881299850 (584h59m21.752382885s) [seg_pos={0 0}, upload=512805]: Done begin upload segment. Dec 30 00:28:21.881324306 (584h59m21.752407341s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.881389700 (584h59m21.752472745s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.882087338 (584h59m21.753170374s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883576975 (584h59m21.75466001s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883586743 (584h59m21.754669779s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884026291 (584h59m21.755109336s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884040708 (584h59m21.755123743s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884417556 (584h59m21.755500601s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884441141 (584h59m21.755524176s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884776419 (584h59m21.755859464s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884844208 (584h59m21.755927243s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885201499 (584h59m21.756284534s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885560672 (584h59m21.756643708s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885971014 (584h59m21.757054069s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.886602937 (584h59m21.757685972s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.887486650 (584h59m21.758569695s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:23.486845051 (584h59m23.357928086s) [upload=512805]: Next returned nil segment Dec 30 00:28:23.486866302 (584h59m23.357949337s) [upload=512805]: Waiting for error group managing segments... Dec 30 00:28:23.868031110 (584h59m23.739114165s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.871946364 (584h59m23.743029399s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874212258 (584h59m23.745295303s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874377704 (584h59m23.745460739s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874851677 (584h59m23.745934712s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.878945862 (584h59m23.750028897s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.881991060 (584h59m23.753074105s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.887715971 (584h59m23.758799006s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.054678917 (584h59m23.925761952s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.109927864 (584h59m23.981010899s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.139250515 (584h59m24.01033355s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.341783439 (584h59m24.212866484s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436862164 (584h59m24.307945199s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436881480 (584h59m24.307964525s) [seg_pos={0 0}, upload=512805]: Segment reached optimal threshold of 13 pieces. Dec 30 00:28:24.437052376 (584h59m24.308135421s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 00:28:24.437058267 (584h59m24.308141312s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437174368 (584h59m24.308257403s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection Dec 30 00:28:24.437182995 (584h59m24.30826604s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437218943 (584h59m24.308301978s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection Dec 30 00:28:24.437224754 (584h59m24.308307789s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437238090 (584h59m24.308321125s) [seg_pos={0 0}, upload=512805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:28:24.437298264 (584h59m24.308381309s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:28:24.437302632 (584h59m24.308385677s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:28:24.726409577 (584h59m24.597492613s) [upload=512805]: Done uploading segments... Dec 30 00:28:24.726519537 (584h59m24.597602572s) [upload=512805]: Upload finished. err: <nil> Duration: 7.683702622s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 28.925097617s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:28:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 28.925119958s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.571345429s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:28:52 UTC) orbiter warning: context deadline exceeded Duration: 3.571374695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.........XXX....X...X..........X......XX.X.............X...X] duration: 30.002566355s trace: https://jaeger.dev.storj.io/trace/4576e0b260f1cccc FAIL (2025-12-30 00:28:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA) context deadline exceeded ======LOGS====== egments... Dec 30 00:28:21.635917127 (584h59m21.507000173s) [upload=512805]: Got next segment. Inline: false Dec 30 00:28:21.635940682 (584h59m21.507023717s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:28:21.635944719 (584h59m21.507027754s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:28:21.881257599 (584h59m21.752340644s) [seg_pos={0 0}, upload=512805]: Begin upload segment... Dec 30 00:28:21.881299850 (584h59m21.752382885s) [seg_pos={0 0}, upload=512805]: Done begin upload segment. Dec 30 00:28:21.881324306 (584h59m21.752407341s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.881389700 (584h59m21.752472745s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.882087338 (584h59m21.753170374s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883576975 (584h59m21.75466001s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.883586743 (584h59m21.754669779s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884026291 (584h59m21.755109336s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884040708 (584h59m21.755123743s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884417556 (584h59m21.755500601s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884441141 (584h59m21.755524176s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884776419 (584h59m21.755859464s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.884844208 (584h59m21.755927243s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885201499 (584h59m21.756284534s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885560672 (584h59m21.756643708s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.885971014 (584h59m21.757054069s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.886602937 (584h59m21.757685972s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:21.887486650 (584h59m21.758569695s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Uploading piece... Dec 30 00:28:23.486845051 (584h59m23.357928086s) [upload=512805]: Next returned nil segment Dec 30 00:28:23.486866302 (584h59m23.357949337s) [upload=512805]: Waiting for error group managing segments... Dec 30 00:28:23.868031110 (584h59m23.739114165s) [noise=true , address=145.79.227.45:4230 , piece_id=343C3LGPKQF7QC32I2IEV2WCVSHI623M53U7JL2UX4FQFDFDGBLA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.871946364 (584h59m23.743029399s) [noise=true , address=145.79.227.18:4007 , piece_id=YN7ZLF7LCOCJGVCLVKRCOLZAEPEK6H35CTHVCBC5F4NWSCVNKO4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874212258 (584h59m23.745295303s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYSA54EDLIKDPQRJ7JL5EQQE4CSRN3ZMDCW5WUJN5YBBDC7IKFA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874377704 (584h59m23.745460739s) [noise=true , address=145.79.227.4:4186 , piece_id=FANGTYB3MVHV2RVLW5SKYVXZQ5OFCSWHI3MGBJS43SCARU6SRNYA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.874851677 (584h59m23.745934712s) [noise=true , address=145.79.227.15:4875 , piece_id=X5AK5GIROJ3SXHHJ7CLTK4VBKDDJWENW3JZZMMRR2WNJDRDFD7DA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.878945862 (584h59m23.750028897s) [noise=true , address=145.79.227.32:4836 , piece_id=VJBGTW4RMYTWFCDWSGR7FG4T3EZFHNFGQKAT2T23VP2HIWN7CTQQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.881991060 (584h59m23.753074105s) [noise=true , address=145.79.227.22:4509 , piece_id=CTBVSZOD43WTZTHFGIT46KRQAFTHHYE7WWRI4L2KEJUFGXHXBQUA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:23.887715971 (584h59m23.758799006s) [noise=true , address=145.79.227.39:4015 , piece_id=BOUXDS6DIYY7NGV7S4WNJQRUBUWKSBZU6LCFNI25SMIFX7DBRJWA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.054678917 (584h59m23.925761952s) [noise=true , address=77.90.31.14:4663 , piece_id=YDWOPINVKMC4EUIQU45DPUM427OVM6N2MC7VYLGXMWIM4XIQX3GQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.109927864 (584h59m23.981010899s) [noise=true , address=77.90.31.17:4773 , piece_id=KYPAN2LJ2I2QUZ7BBFHWWEIPWTU5W4UBSH6UEKIU6IX5BJ2THVCA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.139250515 (584h59m24.01033355s) [noise=true , address=77.90.31.21:4880 , piece_id=2K66WT7GZLRYRY5UCCUQASF526QWVO6ZWCRRTIW77PMR33B2LS2A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.341783439 (584h59m24.212866484s) [noise=true , address=145.79.227.2:4115 , piece_id=P7VF2XRS7G3K5RFT5JRV5DVYYGVH7TPJJCTNFD5KD6BAQYQXEEWQ, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436862164 (584h59m24.307945199s) [noise=true , address=45.91.116.4:4940 , piece_id=ZZ6URIZXT3PBZACOBSP65V73N3M7TXTU5E7NEGIF7PT3CW4CHX2Q, seg_pos={0 0}, upload=512805]: Done uploading piece. err: <nil> Dec 30 00:28:24.436881480 (584h59m24.307964525s) [seg_pos={0 0}, upload=512805]: Segment reached optimal threshold of 13 pieces. Dec 30 00:28:24.437052376 (584h59m24.308135421s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 00:28:24.437058267 (584h59m24.308141312s) [noise=true , address=145.79.227.43:4157 , piece_id=6ZAZP5ATG65372ZSBCXLX5MZWQAEEY5HR665BWR5WNCG5R6FMH4A, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437174368 (584h59m24.308257403s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60528->45.91.116.11:4120: use of closed network connection Dec 30 00:28:24.437182995 (584h59m24.30826604s) [noise=true , address=45.91.116.11:4120 , piece_id=XLM3DIFBVEHSXRFGZOEDVVJFK6ODQQTRDKNSNE2Q7E3HXN3UDMBA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437218943 (584h59m24.308301978s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55824->45.91.116.7:4010: use of closed network connection Dec 30 00:28:24.437224754 (584h59m24.308307789s) [noise=true , address=45.91.116.7:4010 , piece_id=QRPUWJRECOFE2V6QTBV4ZCZWH3EKTZA4UYES4XDEVJHEYQ4G27YA, seg_pos={0 0}, upload=512805]: Long tail cancellation due to optimal threshold Dec 30 00:28:24.437238090 (584h59m24.308321125s) [seg_pos={0 0}, upload=512805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:28:24.437298264 (584h59m24.308381309s) [upload=512805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:28:24.437302632 (584h59m24.308385677s) [upload=512805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:28:24.726409577 (584h59m24.597492613s) [upload=512805]: Done uploading segments... Dec 30 00:28:24.726519537 (584h59m24.597602572s) [upload=512805]: Upload finished. err: <nil> Duration: 30.002587646s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.490795706s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:28:21 UTC) orbiter warning: context deadline exceeded Duration: 3.49083396s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 23.386598005s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:27:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 23.386625266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........XXX....X...X..........X......XX.X.............X...] duration: 5.988743871s trace: https://jaeger.dev.storj.io/trace/2effb61801f693d9 FAIL (2025-12-30 00:27:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :24:17.020913890 (584h55m16.891996935s) [upload=512749]: Starting upload Dec 30 00:24:17.021004343 (584h55m16.892087378s) [upload=512749]: Done starting upload Dec 30 00:24:17.021091578 (584h55m16.892174623s) [upload=512749]: Uploading segments... Dec 30 00:24:17.021311618 (584h55m16.892394673s) [upload=512749]: Got next segment. Inline: false Dec 30 00:24:17.021363055 (584h55m16.892446101s) [upload=512749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:24:17.021370419 (584h55m16.892453454s) [upload=512749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:24:17.269842543 (584h55m17.140925578s) [seg_pos={0 0}, upload=512749]: Begin upload segment... Dec 30 00:24:17.269884935 (584h55m17.14096798s) [seg_pos={0 0}, upload=512749]: Done begin upload segment. Dec 30 00:24:17.269905704 (584h55m17.140988739s) [noise=true , address=145.79.227.10:4402 , piece_id=MYBEB7BZUAD6TCASB4IMFASBA2YXQERGCLOIE6N3XAVOIROB4P4A, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.269923188 (584h55m17.141006223s) [noise=true , address=77.90.31.14:4664 , piece_id=EMAO3IPQXCQ7PCGRXBMNOPUIEDWRGGQHV5MGWW3TWVEOV2P6DCPQ, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.270629352 (584h55m17.141712387s) [noise=true , address=145.79.227.37:4566 , piece_id=UONVOCHJ5SNPSAVEBQPT5K5FZ5XMTLKX2H5I3G2OJMIIUI3DOZWA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.270973377 (584h55m17.142056412s) [noise=true , address=145.79.227.18:4006 , piece_id=V2YTPBO4ISYD3OWHUPN3PY6G5T24DFV7PVMLV2XF2GPX7KX7NSBA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.271467428 (584h55m17.142550473s) [noise=true , address=145.79.227.41:4085 , piece_id=GORS4Y5777L3UIRZZTMWKGS67LSUXN4K2O263KDEFW3QKSUGZ7EQ, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.271899901 (584h55m17.142982936s) [noise=true , address=45.91.116.11:4117 , piece_id=JKX6ZNOVH76G5ILFVW5ZA3G3UQ3DAJGFJYB5UKFOSTG7B3EAWQDA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.272289883 (584h55m17.143372918s) [noise=true , address=45.91.116.7:4012 , piece_id=IJWVDTOFYPEJWON6JUI2Q6BDYCAKMM7USVC4UYGJZQLTQEF3GHUA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.272665689 (584h55m17.143748724s) [noise=true , address=77.90.31.19:4842 , piece_id=IDD4TTGEVEUFUHTAMOKX4BGPWQBNSFO73BCLFXZXQEIXQMV5GG2A, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.273766015 (584h55m17.14484905s) [noise=true , address=145.79.227.6:4258 , piece_id=WQ25S4CABGDHFY73XJCP3H5MCQBZJ4M35KQJRUKPVB7MII4IIYXQ, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.274127002 (584h55m17.145210067s) [noise=true , address=77.90.31.11:4554 , piece_id=O6RO36XVKN2URAQ26LN2GTTE4SVIYXWEJ4NCQNVU5OJUGYQNVTDA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.275626266 (584h55m17.146709301s) [noise=true , address=45.91.116.12:4225 , piece_id=3SOZ3LXABGYA5L3I57RQAGTOZCUSTETN3V4SXVOSUEGI44N76AEA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.276123202 (584h55m17.147206237s) [noise=true , address=145.79.227.22:4509 , piece_id=LWNPFRMZF2PS6JOBEPK3BZ57ZGJQLMK6E4V3EKFR6OJB23MTZD6Q, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.277504663 (584h55m17.148587698s) [noise=true , address=45.91.116.8:4047 , piece_id=V5LYGLJWEBWCI2DGEY2QUAA5SINM5UMJEW4UKCFWQMZCMJNP5U6Q, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.278908165 (584h55m17.1499912s) [noise=true , address=145.79.227.32:4834 , piece_id=5HGOC2JJIKS5XIZQSPAQ7AQGQRCE52ESVOISNP4V4A6F4OBGOK2A, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.280190416 (584h55m17.151273451s) [noise=true , address=145.79.227.36:4517 , piece_id=QQBNOON27G2TVMFQLJ6OHUJWVE2K5P5SIXGAFKLMQJIPH6SBZOWQ, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.281078247 (584h55m17.152161282s) [noise=true , address=145.79.227.51:4411 , piece_id=AAB4H2443U2DYH3ELUQHJZ4MOVDIZVBJFOFYAZJHID262AN4NDOA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:18.932077757 (584h55m18.803160792s) [upload=512749]: Next returned nil segment Dec 30 00:24:18.932094077 (584h55m18.803177123s) [upload=512749]: Waiting for error group managing segments... Dec 30 00:24:19.261365087 (584h55m19.132448122s) [noise=true , address=145.79.227.51:4411 , piece_id=AAB4H2443U2DYH3ELUQHJZ4MOVDIZVBJFOFYAZJHID262AN4NDOA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.262984610 (584h55m19.134067655s) [noise=true , address=145.79.227.18:4006 , piece_id=V2YTPBO4ISYD3OWHUPN3PY6G5T24DFV7PVMLV2XF2GPX7KX7NSBA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.266911958 (584h55m19.137994993s) [noise=true , address=145.79.227.22:4509 , piece_id=LWNPFRMZF2PS6JOBEPK3BZ57ZGJQLMK6E4V3EKFR6OJB23MTZD6Q, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.268020999 (584h55m19.139104054s) [noise=true , address=145.79.227.41:4085 , piece_id=GORS4Y5777L3UIRZZTMWKGS67LSUXN4K2O263KDEFW3QKSUGZ7EQ, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.277124834 (584h55m19.148207869s) [noise=true , address=145.79.227.10:4402 , piece_id=MYBEB7BZUAD6TCASB4IMFASBA2YXQERGCLOIE6N3XAVOIROB4P4A, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.288290614 (584h55m19.159373659s) [noise=true , address=145.79.227.36:4517 , piece_id=QQBNOON27G2TVMFQLJ6OHUJWVE2K5P5SIXGAFKLMQJIPH6SBZOWQ, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.301141182 (584h55m19.172224217s) [noise=true , address=145.79.227.6:4258 , piece_id=WQ25S4CABGDHFY73XJCP3H5MCQBZJ4M35KQJRUKPVB7MII4IIYXQ, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.424396647 (584h55m19.295479682s) [noise=true , address=145.79.227.32:4834 , piece_id=5HGOC2JJIKS5XIZQSPAQ7AQGQRCE52ESVOISNP4V4A6F4OBGOK2A, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.455576465 (584h55m19.326659511s) [noise=true , address=77.90.31.19:4842 , piece_id=IDD4TTGEVEUFUHTAMOKX4BGPWQBNSFO73BCLFXZXQEIXQMV5GG2A, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.483296426 (584h55m19.354379461s) [noise=true , address=45.91.116.11:4117 , piece_id=JKX6ZNOVH76G5ILFVW5ZA3G3UQ3DAJGFJYB5UKFOSTG7B3EAWQDA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.485147280 (584h55m19.356230315s) [noise=true , address=77.90.31.14:4664 , piece_id=EMAO3IPQXCQ7PCGRXBMNOPUIEDWRGGQHV5MGWW3TWVEOV2P6DCPQ, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.509914087 (584h55m19.380997122s) [noise=true , address=77.90.31.11:4554 , piece_id=O6RO36XVKN2URAQ26LN2GTTE4SVIYXWEJ4NCQNVU5OJUGYQNVTDA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.522367560 (584h55m19.393450605s) [noise=true , address=45.91.116.8:4047 , piece_id=V5LYGLJWEBWCI2DGEY2QUAA5SINM5UMJEW4UKCFWQMZCMJNP5U6Q, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.522392787 (584h55m19.393475832s) [seg_pos={0 0}, upload=512749]: Segment reached optimal threshold of 13 pieces. Dec 30 00:24:19.522716132 (584h55m19.393799177s) [noise=true , address=145.79.227.37:4566 , piece_id=UONVOCHJ5SNPSAVEBQPT5K5FZ5XMTLKX2H5I3G2OJMIIUI3DOZWA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: write tcp 188.166.225.170:37690->145.79.227.37:4566: use of closed network connection; write tcp 188.166.225.170:37690->145.79.227.37:4566: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37690->145.79.227.37:4566: use of closed network connection Dec 30 00:24:19.522726693 (584h55m19.393809738s) [noise=true , address=145.79.227.37:4566 , piece_id=UONVOCHJ5SNPSAVEBQPT5K5FZ5XMTLKX2H5I3G2OJMIIUI3DOZWA, seg_pos={0 0}, upload=512749]: Long tail cancellation due to optimal threshold Dec 30 00:24:19.522930702 (584h55m19.394013757s) [noise=true , address=45.91.116.12:4225 , piece_id=3SOZ3LXABGYA5L3I57RQAGTOZCUSTETN3V4SXVOSUEGI44N76AEA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 00:24:19.522939729 (584h55m19.394022764s) [noise=true , address=45.91.116.12:4225 , piece_id=3SOZ3LXABGYA5L3I57RQAGTOZCUSTETN3V4SXVOSUEGI44N76AEA, seg_pos={0 0}, upload=512749]: Long tail cancellation due to optimal threshold Dec 30 00:24:19.522955679 (584h55m19.394038724s) [noise=true , address=45.91.116.7:4012 , piece_id=IJWVDTOFYPEJWON6JUI2Q6BDYCAKMM7USVC4UYGJZQLTQEF3GHUA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 00:24:19.522978142 (584h55m19.394061177s) [noise=true , address=45.91.116.7:4012 , piece_id=IJWVDTOFYPEJWON6JUI2Q6BDYCAKMM7USVC4UYGJZQLTQEF3GHUA, seg_pos={0 0}, upload=512749]: Long tail cancellation due to optimal threshold Dec 30 00:24:19.523009191 (584h55m19.394092226s) [seg_pos={0 0}, upload=512749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:24:19.523131213 (584h55m19.394214248s) [upload=512749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:24:19.523138227 (584h55m19.394221272s) [upload=512749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:24:19.812414253 (584h55m19.683497288s) [upload=512749]: Done uploading segments... Dec 30 00:24:19.812433910 (584h55m19.683516945s) [upload=512749]: Upload finished. err: <nil> Duration: 5.988771773s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.528483809s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:27:17 UTC) orbiter warning: context deadline exceeded Duration: 3.528567638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 20.201717364s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:26:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 20.201738724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 31.29984093s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:26:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 31.299881057s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........XXX....X...X..........X......XX.X.............X..] duration: 6.040832742s trace: https://jaeger.dev.storj.io/trace/2effb61801f693d9 FAIL (2025-12-30 00:26:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :24:17.020913890 (584h55m16.891996935s) [upload=512749]: Starting upload Dec 30 00:24:17.021004343 (584h55m16.892087378s) [upload=512749]: Done starting upload Dec 30 00:24:17.021091578 (584h55m16.892174623s) [upload=512749]: Uploading segments... Dec 30 00:24:17.021311618 (584h55m16.892394673s) [upload=512749]: Got next segment. Inline: false Dec 30 00:24:17.021363055 (584h55m16.892446101s) [upload=512749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:24:17.021370419 (584h55m16.892453454s) [upload=512749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:24:17.269842543 (584h55m17.140925578s) [seg_pos={0 0}, upload=512749]: Begin upload segment... Dec 30 00:24:17.269884935 (584h55m17.14096798s) [seg_pos={0 0}, upload=512749]: Done begin upload segment. Dec 30 00:24:17.269905704 (584h55m17.140988739s) [noise=true , address=145.79.227.10:4402 , piece_id=MYBEB7BZUAD6TCASB4IMFASBA2YXQERGCLOIE6N3XAVOIROB4P4A, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.269923188 (584h55m17.141006223s) [noise=true , address=77.90.31.14:4664 , piece_id=EMAO3IPQXCQ7PCGRXBMNOPUIEDWRGGQHV5MGWW3TWVEOV2P6DCPQ, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.270629352 (584h55m17.141712387s) [noise=true , address=145.79.227.37:4566 , piece_id=UONVOCHJ5SNPSAVEBQPT5K5FZ5XMTLKX2H5I3G2OJMIIUI3DOZWA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.270973377 (584h55m17.142056412s) [noise=true , address=145.79.227.18:4006 , piece_id=V2YTPBO4ISYD3OWHUPN3PY6G5T24DFV7PVMLV2XF2GPX7KX7NSBA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.271467428 (584h55m17.142550473s) [noise=true , address=145.79.227.41:4085 , piece_id=GORS4Y5777L3UIRZZTMWKGS67LSUXN4K2O263KDEFW3QKSUGZ7EQ, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.271899901 (584h55m17.142982936s) [noise=true , address=45.91.116.11:4117 , piece_id=JKX6ZNOVH76G5ILFVW5ZA3G3UQ3DAJGFJYB5UKFOSTG7B3EAWQDA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.272289883 (584h55m17.143372918s) [noise=true , address=45.91.116.7:4012 , piece_id=IJWVDTOFYPEJWON6JUI2Q6BDYCAKMM7USVC4UYGJZQLTQEF3GHUA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.272665689 (584h55m17.143748724s) [noise=true , address=77.90.31.19:4842 , piece_id=IDD4TTGEVEUFUHTAMOKX4BGPWQBNSFO73BCLFXZXQEIXQMV5GG2A, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.273766015 (584h55m17.14484905s) [noise=true , address=145.79.227.6:4258 , piece_id=WQ25S4CABGDHFY73XJCP3H5MCQBZJ4M35KQJRUKPVB7MII4IIYXQ, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.274127002 (584h55m17.145210067s) [noise=true , address=77.90.31.11:4554 , piece_id=O6RO36XVKN2URAQ26LN2GTTE4SVIYXWEJ4NCQNVU5OJUGYQNVTDA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.275626266 (584h55m17.146709301s) [noise=true , address=45.91.116.12:4225 , piece_id=3SOZ3LXABGYA5L3I57RQAGTOZCUSTETN3V4SXVOSUEGI44N76AEA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.276123202 (584h55m17.147206237s) [noise=true , address=145.79.227.22:4509 , piece_id=LWNPFRMZF2PS6JOBEPK3BZ57ZGJQLMK6E4V3EKFR6OJB23MTZD6Q, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.277504663 (584h55m17.148587698s) [noise=true , address=45.91.116.8:4047 , piece_id=V5LYGLJWEBWCI2DGEY2QUAA5SINM5UMJEW4UKCFWQMZCMJNP5U6Q, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.278908165 (584h55m17.1499912s) [noise=true , address=145.79.227.32:4834 , piece_id=5HGOC2JJIKS5XIZQSPAQ7AQGQRCE52ESVOISNP4V4A6F4OBGOK2A, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.280190416 (584h55m17.151273451s) [noise=true , address=145.79.227.36:4517 , piece_id=QQBNOON27G2TVMFQLJ6OHUJWVE2K5P5SIXGAFKLMQJIPH6SBZOWQ, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.281078247 (584h55m17.152161282s) [noise=true , address=145.79.227.51:4411 , piece_id=AAB4H2443U2DYH3ELUQHJZ4MOVDIZVBJFOFYAZJHID262AN4NDOA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:18.932077757 (584h55m18.803160792s) [upload=512749]: Next returned nil segment Dec 30 00:24:18.932094077 (584h55m18.803177123s) [upload=512749]: Waiting for error group managing segments... Dec 30 00:24:19.261365087 (584h55m19.132448122s) [noise=true , address=145.79.227.51:4411 , piece_id=AAB4H2443U2DYH3ELUQHJZ4MOVDIZVBJFOFYAZJHID262AN4NDOA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.262984610 (584h55m19.134067655s) [noise=true , address=145.79.227.18:4006 , piece_id=V2YTPBO4ISYD3OWHUPN3PY6G5T24DFV7PVMLV2XF2GPX7KX7NSBA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.266911958 (584h55m19.137994993s) [noise=true , address=145.79.227.22:4509 , piece_id=LWNPFRMZF2PS6JOBEPK3BZ57ZGJQLMK6E4V3EKFR6OJB23MTZD6Q, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.268020999 (584h55m19.139104054s) [noise=true , address=145.79.227.41:4085 , piece_id=GORS4Y5777L3UIRZZTMWKGS67LSUXN4K2O263KDEFW3QKSUGZ7EQ, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.277124834 (584h55m19.148207869s) [noise=true , address=145.79.227.10:4402 , piece_id=MYBEB7BZUAD6TCASB4IMFASBA2YXQERGCLOIE6N3XAVOIROB4P4A, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.288290614 (584h55m19.159373659s) [noise=true , address=145.79.227.36:4517 , piece_id=QQBNOON27G2TVMFQLJ6OHUJWVE2K5P5SIXGAFKLMQJIPH6SBZOWQ, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.301141182 (584h55m19.172224217s) [noise=true , address=145.79.227.6:4258 , piece_id=WQ25S4CABGDHFY73XJCP3H5MCQBZJ4M35KQJRUKPVB7MII4IIYXQ, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.424396647 (584h55m19.295479682s) [noise=true , address=145.79.227.32:4834 , piece_id=5HGOC2JJIKS5XIZQSPAQ7AQGQRCE52ESVOISNP4V4A6F4OBGOK2A, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.455576465 (584h55m19.326659511s) [noise=true , address=77.90.31.19:4842 , piece_id=IDD4TTGEVEUFUHTAMOKX4BGPWQBNSFO73BCLFXZXQEIXQMV5GG2A, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.483296426 (584h55m19.354379461s) [noise=true , address=45.91.116.11:4117 , piece_id=JKX6ZNOVH76G5ILFVW5ZA3G3UQ3DAJGFJYB5UKFOSTG7B3EAWQDA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.485147280 (584h55m19.356230315s) [noise=true , address=77.90.31.14:4664 , piece_id=EMAO3IPQXCQ7PCGRXBMNOPUIEDWRGGQHV5MGWW3TWVEOV2P6DCPQ, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.509914087 (584h55m19.380997122s) [noise=true , address=77.90.31.11:4554 , piece_id=O6RO36XVKN2URAQ26LN2GTTE4SVIYXWEJ4NCQNVU5OJUGYQNVTDA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.522367560 (584h55m19.393450605s) [noise=true , address=45.91.116.8:4047 , piece_id=V5LYGLJWEBWCI2DGEY2QUAA5SINM5UMJEW4UKCFWQMZCMJNP5U6Q, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.522392787 (584h55m19.393475832s) [seg_pos={0 0}, upload=512749]: Segment reached optimal threshold of 13 pieces. Dec 30 00:24:19.522716132 (584h55m19.393799177s) [noise=true , address=145.79.227.37:4566 , piece_id=UONVOCHJ5SNPSAVEBQPT5K5FZ5XMTLKX2H5I3G2OJMIIUI3DOZWA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: write tcp 188.166.225.170:37690->145.79.227.37:4566: use of closed network connection; write tcp 188.166.225.170:37690->145.79.227.37:4566: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37690->145.79.227.37:4566: use of closed network connection Dec 30 00:24:19.522726693 (584h55m19.393809738s) [noise=true , address=145.79.227.37:4566 , piece_id=UONVOCHJ5SNPSAVEBQPT5K5FZ5XMTLKX2H5I3G2OJMIIUI3DOZWA, seg_pos={0 0}, upload=512749]: Long tail cancellation due to optimal threshold Dec 30 00:24:19.522930702 (584h55m19.394013757s) [noise=true , address=45.91.116.12:4225 , piece_id=3SOZ3LXABGYA5L3I57RQAGTOZCUSTETN3V4SXVOSUEGI44N76AEA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 00:24:19.522939729 (584h55m19.394022764s) [noise=true , address=45.91.116.12:4225 , piece_id=3SOZ3LXABGYA5L3I57RQAGTOZCUSTETN3V4SXVOSUEGI44N76AEA, seg_pos={0 0}, upload=512749]: Long tail cancellation due to optimal threshold Dec 30 00:24:19.522955679 (584h55m19.394038724s) [noise=true , address=45.91.116.7:4012 , piece_id=IJWVDTOFYPEJWON6JUI2Q6BDYCAKMM7USVC4UYGJZQLTQEF3GHUA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 00:24:19.522978142 (584h55m19.394061177s) [noise=true , address=45.91.116.7:4012 , piece_id=IJWVDTOFYPEJWON6JUI2Q6BDYCAKMM7USVC4UYGJZQLTQEF3GHUA, seg_pos={0 0}, upload=512749]: Long tail cancellation due to optimal threshold Dec 30 00:24:19.523009191 (584h55m19.394092226s) [seg_pos={0 0}, upload=512749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:24:19.523131213 (584h55m19.394214248s) [upload=512749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:24:19.523138227 (584h55m19.394221272s) [upload=512749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:24:19.812414253 (584h55m19.683497288s) [upload=512749]: Done uploading segments... Dec 30 00:24:19.812433910 (584h55m19.683516945s) [upload=512749]: Upload finished. err: <nil> Duration: 6.040864662s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.508355802s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:26:05 UTC) orbiter warning: context deadline exceeded Duration: 3.508394415s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.53085998s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:25:11 UTC) orbiter warning: context deadline exceeded Duration: 3.530887464s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [............XXX....X...X..........X......XX.X.............X.] duration: 5.958438818s trace: https://jaeger.dev.storj.io/trace/2effb61801f693d9 FAIL (2025-12-30 00:25:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :24:17.020913890 (584h55m16.891996935s) [upload=512749]: Starting upload Dec 30 00:24:17.021004343 (584h55m16.892087378s) [upload=512749]: Done starting upload Dec 30 00:24:17.021091578 (584h55m16.892174623s) [upload=512749]: Uploading segments... Dec 30 00:24:17.021311618 (584h55m16.892394673s) [upload=512749]: Got next segment. Inline: false Dec 30 00:24:17.021363055 (584h55m16.892446101s) [upload=512749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:24:17.021370419 (584h55m16.892453454s) [upload=512749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:24:17.269842543 (584h55m17.140925578s) [seg_pos={0 0}, upload=512749]: Begin upload segment... Dec 30 00:24:17.269884935 (584h55m17.14096798s) [seg_pos={0 0}, upload=512749]: Done begin upload segment. Dec 30 00:24:17.269905704 (584h55m17.140988739s) [noise=true , address=145.79.227.10:4402 , piece_id=MYBEB7BZUAD6TCASB4IMFASBA2YXQERGCLOIE6N3XAVOIROB4P4A, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.269923188 (584h55m17.141006223s) [noise=true , address=77.90.31.14:4664 , piece_id=EMAO3IPQXCQ7PCGRXBMNOPUIEDWRGGQHV5MGWW3TWVEOV2P6DCPQ, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.270629352 (584h55m17.141712387s) [noise=true , address=145.79.227.37:4566 , piece_id=UONVOCHJ5SNPSAVEBQPT5K5FZ5XMTLKX2H5I3G2OJMIIUI3DOZWA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.270973377 (584h55m17.142056412s) [noise=true , address=145.79.227.18:4006 , piece_id=V2YTPBO4ISYD3OWHUPN3PY6G5T24DFV7PVMLV2XF2GPX7KX7NSBA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.271467428 (584h55m17.142550473s) [noise=true , address=145.79.227.41:4085 , piece_id=GORS4Y5777L3UIRZZTMWKGS67LSUXN4K2O263KDEFW3QKSUGZ7EQ, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.271899901 (584h55m17.142982936s) [noise=true , address=45.91.116.11:4117 , piece_id=JKX6ZNOVH76G5ILFVW5ZA3G3UQ3DAJGFJYB5UKFOSTG7B3EAWQDA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.272289883 (584h55m17.143372918s) [noise=true , address=45.91.116.7:4012 , piece_id=IJWVDTOFYPEJWON6JUI2Q6BDYCAKMM7USVC4UYGJZQLTQEF3GHUA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.272665689 (584h55m17.143748724s) [noise=true , address=77.90.31.19:4842 , piece_id=IDD4TTGEVEUFUHTAMOKX4BGPWQBNSFO73BCLFXZXQEIXQMV5GG2A, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.273766015 (584h55m17.14484905s) [noise=true , address=145.79.227.6:4258 , piece_id=WQ25S4CABGDHFY73XJCP3H5MCQBZJ4M35KQJRUKPVB7MII4IIYXQ, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.274127002 (584h55m17.145210067s) [noise=true , address=77.90.31.11:4554 , piece_id=O6RO36XVKN2URAQ26LN2GTTE4SVIYXWEJ4NCQNVU5OJUGYQNVTDA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.275626266 (584h55m17.146709301s) [noise=true , address=45.91.116.12:4225 , piece_id=3SOZ3LXABGYA5L3I57RQAGTOZCUSTETN3V4SXVOSUEGI44N76AEA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.276123202 (584h55m17.147206237s) [noise=true , address=145.79.227.22:4509 , piece_id=LWNPFRMZF2PS6JOBEPK3BZ57ZGJQLMK6E4V3EKFR6OJB23MTZD6Q, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.277504663 (584h55m17.148587698s) [noise=true , address=45.91.116.8:4047 , piece_id=V5LYGLJWEBWCI2DGEY2QUAA5SINM5UMJEW4UKCFWQMZCMJNP5U6Q, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.278908165 (584h55m17.1499912s) [noise=true , address=145.79.227.32:4834 , piece_id=5HGOC2JJIKS5XIZQSPAQ7AQGQRCE52ESVOISNP4V4A6F4OBGOK2A, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.280190416 (584h55m17.151273451s) [noise=true , address=145.79.227.36:4517 , piece_id=QQBNOON27G2TVMFQLJ6OHUJWVE2K5P5SIXGAFKLMQJIPH6SBZOWQ, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.281078247 (584h55m17.152161282s) [noise=true , address=145.79.227.51:4411 , piece_id=AAB4H2443U2DYH3ELUQHJZ4MOVDIZVBJFOFYAZJHID262AN4NDOA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:18.932077757 (584h55m18.803160792s) [upload=512749]: Next returned nil segment Dec 30 00:24:18.932094077 (584h55m18.803177123s) [upload=512749]: Waiting for error group managing segments... Dec 30 00:24:19.261365087 (584h55m19.132448122s) [noise=true , address=145.79.227.51:4411 , piece_id=AAB4H2443U2DYH3ELUQHJZ4MOVDIZVBJFOFYAZJHID262AN4NDOA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.262984610 (584h55m19.134067655s) [noise=true , address=145.79.227.18:4006 , piece_id=V2YTPBO4ISYD3OWHUPN3PY6G5T24DFV7PVMLV2XF2GPX7KX7NSBA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.266911958 (584h55m19.137994993s) [noise=true , address=145.79.227.22:4509 , piece_id=LWNPFRMZF2PS6JOBEPK3BZ57ZGJQLMK6E4V3EKFR6OJB23MTZD6Q, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.268020999 (584h55m19.139104054s) [noise=true , address=145.79.227.41:4085 , piece_id=GORS4Y5777L3UIRZZTMWKGS67LSUXN4K2O263KDEFW3QKSUGZ7EQ, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.277124834 (584h55m19.148207869s) [noise=true , address=145.79.227.10:4402 , piece_id=MYBEB7BZUAD6TCASB4IMFASBA2YXQERGCLOIE6N3XAVOIROB4P4A, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.288290614 (584h55m19.159373659s) [noise=true , address=145.79.227.36:4517 , piece_id=QQBNOON27G2TVMFQLJ6OHUJWVE2K5P5SIXGAFKLMQJIPH6SBZOWQ, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.301141182 (584h55m19.172224217s) [noise=true , address=145.79.227.6:4258 , piece_id=WQ25S4CABGDHFY73XJCP3H5MCQBZJ4M35KQJRUKPVB7MII4IIYXQ, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.424396647 (584h55m19.295479682s) [noise=true , address=145.79.227.32:4834 , piece_id=5HGOC2JJIKS5XIZQSPAQ7AQGQRCE52ESVOISNP4V4A6F4OBGOK2A, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.455576465 (584h55m19.326659511s) [noise=true , address=77.90.31.19:4842 , piece_id=IDD4TTGEVEUFUHTAMOKX4BGPWQBNSFO73BCLFXZXQEIXQMV5GG2A, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.483296426 (584h55m19.354379461s) [noise=true , address=45.91.116.11:4117 , piece_id=JKX6ZNOVH76G5ILFVW5ZA3G3UQ3DAJGFJYB5UKFOSTG7B3EAWQDA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.485147280 (584h55m19.356230315s) [noise=true , address=77.90.31.14:4664 , piece_id=EMAO3IPQXCQ7PCGRXBMNOPUIEDWRGGQHV5MGWW3TWVEOV2P6DCPQ, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.509914087 (584h55m19.380997122s) [noise=true , address=77.90.31.11:4554 , piece_id=O6RO36XVKN2URAQ26LN2GTTE4SVIYXWEJ4NCQNVU5OJUGYQNVTDA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.522367560 (584h55m19.393450605s) [noise=true , address=45.91.116.8:4047 , piece_id=V5LYGLJWEBWCI2DGEY2QUAA5SINM5UMJEW4UKCFWQMZCMJNP5U6Q, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.522392787 (584h55m19.393475832s) [seg_pos={0 0}, upload=512749]: Segment reached optimal threshold of 13 pieces. Dec 30 00:24:19.522716132 (584h55m19.393799177s) [noise=true , address=145.79.227.37:4566 , piece_id=UONVOCHJ5SNPSAVEBQPT5K5FZ5XMTLKX2H5I3G2OJMIIUI3DOZWA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: write tcp 188.166.225.170:37690->145.79.227.37:4566: use of closed network connection; write tcp 188.166.225.170:37690->145.79.227.37:4566: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37690->145.79.227.37:4566: use of closed network connection Dec 30 00:24:19.522726693 (584h55m19.393809738s) [noise=true , address=145.79.227.37:4566 , piece_id=UONVOCHJ5SNPSAVEBQPT5K5FZ5XMTLKX2H5I3G2OJMIIUI3DOZWA, seg_pos={0 0}, upload=512749]: Long tail cancellation due to optimal threshold Dec 30 00:24:19.522930702 (584h55m19.394013757s) [noise=true , address=45.91.116.12:4225 , piece_id=3SOZ3LXABGYA5L3I57RQAGTOZCUSTETN3V4SXVOSUEGI44N76AEA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 00:24:19.522939729 (584h55m19.394022764s) [noise=true , address=45.91.116.12:4225 , piece_id=3SOZ3LXABGYA5L3I57RQAGTOZCUSTETN3V4SXVOSUEGI44N76AEA, seg_pos={0 0}, upload=512749]: Long tail cancellation due to optimal threshold Dec 30 00:24:19.522955679 (584h55m19.394038724s) [noise=true , address=45.91.116.7:4012 , piece_id=IJWVDTOFYPEJWON6JUI2Q6BDYCAKMM7USVC4UYGJZQLTQEF3GHUA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 00:24:19.522978142 (584h55m19.394061177s) [noise=true , address=45.91.116.7:4012 , piece_id=IJWVDTOFYPEJWON6JUI2Q6BDYCAKMM7USVC4UYGJZQLTQEF3GHUA, seg_pos={0 0}, upload=512749]: Long tail cancellation due to optimal threshold Dec 30 00:24:19.523009191 (584h55m19.394092226s) [seg_pos={0 0}, upload=512749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:24:19.523131213 (584h55m19.394214248s) [upload=512749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:24:19.523138227 (584h55m19.394221272s) [upload=512749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:24:19.812414253 (584h55m19.683497288s) [upload=512749]: Done uploading segments... Dec 30 00:24:19.812433910 (584h55m19.683516945s) [upload=512749]: Upload finished. err: <nil> Duration: 5.95857036s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 15.408820287s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:25:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 15.408854622s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X............XXX....X...X..........X......XX.X.............X] duration: 30.002097839s trace: https://jaeger.dev.storj.io/trace/2effb61801f693d9 FAIL (2025-12-30 00:24:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :24:17.020913890 (584h55m16.891996935s) [upload=512749]: Starting upload Dec 30 00:24:17.021004343 (584h55m16.892087378s) [upload=512749]: Done starting upload Dec 30 00:24:17.021091578 (584h55m16.892174623s) [upload=512749]: Uploading segments... Dec 30 00:24:17.021311618 (584h55m16.892394673s) [upload=512749]: Got next segment. Inline: false Dec 30 00:24:17.021363055 (584h55m16.892446101s) [upload=512749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:24:17.021370419 (584h55m16.892453454s) [upload=512749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:24:17.269842543 (584h55m17.140925578s) [seg_pos={0 0}, upload=512749]: Begin upload segment... Dec 30 00:24:17.269884935 (584h55m17.14096798s) [seg_pos={0 0}, upload=512749]: Done begin upload segment. Dec 30 00:24:17.269905704 (584h55m17.140988739s) [noise=true , address=145.79.227.10:4402 , piece_id=MYBEB7BZUAD6TCASB4IMFASBA2YXQERGCLOIE6N3XAVOIROB4P4A, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.269923188 (584h55m17.141006223s) [noise=true , address=77.90.31.14:4664 , piece_id=EMAO3IPQXCQ7PCGRXBMNOPUIEDWRGGQHV5MGWW3TWVEOV2P6DCPQ, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.270629352 (584h55m17.141712387s) [noise=true , address=145.79.227.37:4566 , piece_id=UONVOCHJ5SNPSAVEBQPT5K5FZ5XMTLKX2H5I3G2OJMIIUI3DOZWA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.270973377 (584h55m17.142056412s) [noise=true , address=145.79.227.18:4006 , piece_id=V2YTPBO4ISYD3OWHUPN3PY6G5T24DFV7PVMLV2XF2GPX7KX7NSBA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.271467428 (584h55m17.142550473s) [noise=true , address=145.79.227.41:4085 , piece_id=GORS4Y5777L3UIRZZTMWKGS67LSUXN4K2O263KDEFW3QKSUGZ7EQ, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.271899901 (584h55m17.142982936s) [noise=true , address=45.91.116.11:4117 , piece_id=JKX6ZNOVH76G5ILFVW5ZA3G3UQ3DAJGFJYB5UKFOSTG7B3EAWQDA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.272289883 (584h55m17.143372918s) [noise=true , address=45.91.116.7:4012 , piece_id=IJWVDTOFYPEJWON6JUI2Q6BDYCAKMM7USVC4UYGJZQLTQEF3GHUA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.272665689 (584h55m17.143748724s) [noise=true , address=77.90.31.19:4842 , piece_id=IDD4TTGEVEUFUHTAMOKX4BGPWQBNSFO73BCLFXZXQEIXQMV5GG2A, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.273766015 (584h55m17.14484905s) [noise=true , address=145.79.227.6:4258 , piece_id=WQ25S4CABGDHFY73XJCP3H5MCQBZJ4M35KQJRUKPVB7MII4IIYXQ, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.274127002 (584h55m17.145210067s) [noise=true , address=77.90.31.11:4554 , piece_id=O6RO36XVKN2URAQ26LN2GTTE4SVIYXWEJ4NCQNVU5OJUGYQNVTDA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.275626266 (584h55m17.146709301s) [noise=true , address=45.91.116.12:4225 , piece_id=3SOZ3LXABGYA5L3I57RQAGTOZCUSTETN3V4SXVOSUEGI44N76AEA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.276123202 (584h55m17.147206237s) [noise=true , address=145.79.227.22:4509 , piece_id=LWNPFRMZF2PS6JOBEPK3BZ57ZGJQLMK6E4V3EKFR6OJB23MTZD6Q, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.277504663 (584h55m17.148587698s) [noise=true , address=45.91.116.8:4047 , piece_id=V5LYGLJWEBWCI2DGEY2QUAA5SINM5UMJEW4UKCFWQMZCMJNP5U6Q, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.278908165 (584h55m17.1499912s) [noise=true , address=145.79.227.32:4834 , piece_id=5HGOC2JJIKS5XIZQSPAQ7AQGQRCE52ESVOISNP4V4A6F4OBGOK2A, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.280190416 (584h55m17.151273451s) [noise=true , address=145.79.227.36:4517 , piece_id=QQBNOON27G2TVMFQLJ6OHUJWVE2K5P5SIXGAFKLMQJIPH6SBZOWQ, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:17.281078247 (584h55m17.152161282s) [noise=true , address=145.79.227.51:4411 , piece_id=AAB4H2443U2DYH3ELUQHJZ4MOVDIZVBJFOFYAZJHID262AN4NDOA, seg_pos={0 0}, upload=512749]: Uploading piece... Dec 30 00:24:18.932077757 (584h55m18.803160792s) [upload=512749]: Next returned nil segment Dec 30 00:24:18.932094077 (584h55m18.803177123s) [upload=512749]: Waiting for error group managing segments... Dec 30 00:24:19.261365087 (584h55m19.132448122s) [noise=true , address=145.79.227.51:4411 , piece_id=AAB4H2443U2DYH3ELUQHJZ4MOVDIZVBJFOFYAZJHID262AN4NDOA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.262984610 (584h55m19.134067655s) [noise=true , address=145.79.227.18:4006 , piece_id=V2YTPBO4ISYD3OWHUPN3PY6G5T24DFV7PVMLV2XF2GPX7KX7NSBA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.266911958 (584h55m19.137994993s) [noise=true , address=145.79.227.22:4509 , piece_id=LWNPFRMZF2PS6JOBEPK3BZ57ZGJQLMK6E4V3EKFR6OJB23MTZD6Q, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.268020999 (584h55m19.139104054s) [noise=true , address=145.79.227.41:4085 , piece_id=GORS4Y5777L3UIRZZTMWKGS67LSUXN4K2O263KDEFW3QKSUGZ7EQ, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.277124834 (584h55m19.148207869s) [noise=true , address=145.79.227.10:4402 , piece_id=MYBEB7BZUAD6TCASB4IMFASBA2YXQERGCLOIE6N3XAVOIROB4P4A, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.288290614 (584h55m19.159373659s) [noise=true , address=145.79.227.36:4517 , piece_id=QQBNOON27G2TVMFQLJ6OHUJWVE2K5P5SIXGAFKLMQJIPH6SBZOWQ, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.301141182 (584h55m19.172224217s) [noise=true , address=145.79.227.6:4258 , piece_id=WQ25S4CABGDHFY73XJCP3H5MCQBZJ4M35KQJRUKPVB7MII4IIYXQ, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.424396647 (584h55m19.295479682s) [noise=true , address=145.79.227.32:4834 , piece_id=5HGOC2JJIKS5XIZQSPAQ7AQGQRCE52ESVOISNP4V4A6F4OBGOK2A, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.455576465 (584h55m19.326659511s) [noise=true , address=77.90.31.19:4842 , piece_id=IDD4TTGEVEUFUHTAMOKX4BGPWQBNSFO73BCLFXZXQEIXQMV5GG2A, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.483296426 (584h55m19.354379461s) [noise=true , address=45.91.116.11:4117 , piece_id=JKX6ZNOVH76G5ILFVW5ZA3G3UQ3DAJGFJYB5UKFOSTG7B3EAWQDA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.485147280 (584h55m19.356230315s) [noise=true , address=77.90.31.14:4664 , piece_id=EMAO3IPQXCQ7PCGRXBMNOPUIEDWRGGQHV5MGWW3TWVEOV2P6DCPQ, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.509914087 (584h55m19.380997122s) [noise=true , address=77.90.31.11:4554 , piece_id=O6RO36XVKN2URAQ26LN2GTTE4SVIYXWEJ4NCQNVU5OJUGYQNVTDA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.522367560 (584h55m19.393450605s) [noise=true , address=45.91.116.8:4047 , piece_id=V5LYGLJWEBWCI2DGEY2QUAA5SINM5UMJEW4UKCFWQMZCMJNP5U6Q, seg_pos={0 0}, upload=512749]: Done uploading piece. err: <nil> Dec 30 00:24:19.522392787 (584h55m19.393475832s) [seg_pos={0 0}, upload=512749]: Segment reached optimal threshold of 13 pieces. Dec 30 00:24:19.522716132 (584h55m19.393799177s) [noise=true , address=145.79.227.37:4566 , piece_id=UONVOCHJ5SNPSAVEBQPT5K5FZ5XMTLKX2H5I3G2OJMIIUI3DOZWA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: write tcp 188.166.225.170:37690->145.79.227.37:4566: use of closed network connection; write tcp 188.166.225.170:37690->145.79.227.37:4566: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37690->145.79.227.37:4566: use of closed network connection Dec 30 00:24:19.522726693 (584h55m19.393809738s) [noise=true , address=145.79.227.37:4566 , piece_id=UONVOCHJ5SNPSAVEBQPT5K5FZ5XMTLKX2H5I3G2OJMIIUI3DOZWA, seg_pos={0 0}, upload=512749]: Long tail cancellation due to optimal threshold Dec 30 00:24:19.522930702 (584h55m19.394013757s) [noise=true , address=45.91.116.12:4225 , piece_id=3SOZ3LXABGYA5L3I57RQAGTOZCUSTETN3V4SXVOSUEGI44N76AEA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 00:24:19.522939729 (584h55m19.394022764s) [noise=true , address=45.91.116.12:4225 , piece_id=3SOZ3LXABGYA5L3I57RQAGTOZCUSTETN3V4SXVOSUEGI44N76AEA, seg_pos={0 0}, upload=512749]: Long tail cancellation due to optimal threshold Dec 30 00:24:19.522955679 (584h55m19.394038724s) [noise=true , address=45.91.116.7:4012 , piece_id=IJWVDTOFYPEJWON6JUI2Q6BDYCAKMM7USVC4UYGJZQLTQEF3GHUA, seg_pos={0 0}, upload=512749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 00:24:19.522978142 (584h55m19.394061177s) [noise=true , address=45.91.116.7:4012 , piece_id=IJWVDTOFYPEJWON6JUI2Q6BDYCAKMM7USVC4UYGJZQLTQEF3GHUA, seg_pos={0 0}, upload=512749]: Long tail cancellation due to optimal threshold Dec 30 00:24:19.523009191 (584h55m19.394092226s) [seg_pos={0 0}, upload=512749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:24:19.523131213 (584h55m19.394214248s) [upload=512749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:24:19.523138227 (584h55m19.394221272s) [upload=512749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:24:19.812414253 (584h55m19.683497288s) [upload=512749]: Done uploading segments... Dec 30 00:24:19.812433910 (584h55m19.683516945s) [upload=512749]: Upload finished. err: <nil> Duration: 30.002118719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 36.536416294s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:24:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 36.536442864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.512258594s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:24:01 UTC) orbiter warning: context deadline exceeded Duration: 3.512285996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XX............XXX....X...X..........X......XX.X.............] duration: 6.310439787s trace: https://jaeger.dev.storj.io/trace/1f21e11794748e52 FAIL (2025-12-30 00:23:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: KMYYROA46XZ2QQLP77JLXL5M4BQZDYZUDFPFB36YUAJGPS3XJCLQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: UI4XFR5ED5JU7RJMYJMH35CZPAM462AYTMAAK4M5YPCUJZASZV3A) context deadline exceeded ======LOGS====== 30 00:13:54.324867912 (584h44m54.195950957s) [upload=512597]: Starting upload Dec 30 00:13:54.324990224 (584h44m54.196073259s) [upload=512597]: Done starting upload Dec 30 00:13:54.325054557 (584h44m54.196137602s) [upload=512597]: Uploading segments... Dec 30 00:13:54.325419562 (584h44m54.196502607s) [upload=512597]: Got next segment. Inline: false Dec 30 00:13:54.325469307 (584h44m54.196552362s) [upload=512597]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:13:54.325475618 (584h44m54.196558664s) [upload=512597]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:13:54.580461825 (584h44m54.45154487s) [seg_pos={0 0}, upload=512597]: Begin upload segment... Dec 30 00:13:54.580507753 (584h44m54.451590798s) [seg_pos={0 0}, upload=512597]: Done begin upload segment. Dec 30 00:13:54.580544733 (584h44m54.451627768s) [noise=true , address=145.79.227.37:4567 , piece_id=QBSRM467II7QRO73BZMG5YCI5DGBJ3YJPKTGA4JTNFW7MVE4GCNA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.580601511 (584h44m54.451684556s) [noise=true , address=45.91.116.12:4225 , piece_id=KMYYROA46XZ2QQLP77JLXL5M4BQZDYZUDFPFB36YUAJGPS3XJCLQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.581263281 (584h44m54.452346327s) [noise=true , address=145.79.227.22:4509 , piece_id=IAFA5FPN2CO2EGSYPDH3ZNLMHOF4VIMDWNKPF4JMWQ7UDX7UGMKQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.581718839 (584h44m54.452801874s) [noise=true , address=145.79.227.32:4836 , piece_id=DRKH7VQJJIBS23UBARHVQKHZ2XMAH24GVRDLPNXWBMOZSQAJEHDQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.582118329 (584h44m54.453201364s) [noise=true , address=45.91.116.6:4975 , piece_id=UI4XFR5ED5JU7RJMYJMH35CZPAM462AYTMAAK4M5YPCUJZASZV3A, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.582437077 (584h44m54.453520112s) [noise=true , address=45.91.116.13:4154 , piece_id=RDETXZUSPHCNDATGFT25HGLHJMAA54TU7Z5GTHGOAICPCLX5AJYA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.582894558 (584h44m54.453977603s) [noise=true , address=77.90.31.11:4554 , piece_id=F55TEVYP4YR6CSQCBXCDLZAAZIKHBFLHWIEAT4K4JJ3IDEXHVXPQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.583261416 (584h44m54.454344461s) [noise=true , address=145.79.227.15:4874 , piece_id=H5NXNYBGQFR2IIIAY4T4IOEYFDWLVYQETIJWYJJ4HXI43GEESIDQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.583612384 (584h44m54.454695459s) [noise=true , address=145.79.227.29:4941 , piece_id=ZFKGRHKZ7ZJ75E2S7PBT4NTFDEAIN5AGKFVDC6VDEW7WIGLDRE3Q, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.585392814 (584h44m54.456475859s) [noise=true , address=145.79.227.6:4257 , piece_id=OINK3GXZ3LPJNOEGVQIIM4RQ47CLOB26OAC2ORRP2TBSMWVOMT7A, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.585658430 (584h44m54.456741465s) [noise=true , address=145.79.227.38:4978 , piece_id=T7QMNGNVNTJWXV7IFJVJYSJ4V254FABPDNQXP4OF2HINSJ4W37FA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.587224062 (584h44m54.458307097s) [noise=true , address=145.79.227.47:4266 , piece_id=FABBIGDGNGLNPUAKV3AWMUJWLBADI24JZU57YD3NVONNK6GJZOBQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.588454895 (584h44m54.45953793s) [noise=true , address=77.90.31.19:4841 , piece_id=YW2UDCQC7H7GGHJ5JYITHI73INFNZC5Q4GXE7DYZNAZAO45QW73A, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.589580087 (584h44m54.460663132s) [noise=true , address=45.91.116.14:4189 , piece_id=RLVQQCO37ZG3RN742DKERZDNN6ALSQGTUZAFBW4SZUF2UOBSK6KA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.591136290 (584h44m54.462219325s) [noise=true , address=145.79.227.46:4601 , piece_id=FKFRMZT7M3JS4QBY7GZVM5RFBZH26ZSEPWUWJAW4Q5PNIRQOZZJA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.592682314 (584h44m54.463765359s) [noise=true , address=145.79.227.7:4295 , piece_id=645LDS7KTXFS6WBULVAJK4JF47LGQMMJARTBBA56SFWOX4JP34EA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:56.281718857 (584h44m56.152801902s) [upload=512597]: Next returned nil segment Dec 30 00:13:56.281741130 (584h44m56.152824175s) [upload=512597]: Waiting for error group managing segments... Dec 30 00:13:56.563110793 (584h44m56.434193828s) [noise=true , address=145.79.227.15:4874 , piece_id=H5NXNYBGQFR2IIIAY4T4IOEYFDWLVYQETIJWYJJ4HXI43GEESIDQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.570998270 (584h44m56.442081315s) [noise=true , address=145.79.227.22:4509 , piece_id=IAFA5FPN2CO2EGSYPDH3ZNLMHOF4VIMDWNKPF4JMWQ7UDX7UGMKQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.572245676 (584h44m56.443328721s) [noise=true , address=145.79.227.37:4567 , piece_id=QBSRM467II7QRO73BZMG5YCI5DGBJ3YJPKTGA4JTNFW7MVE4GCNA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.579941789 (584h44m56.451024824s) [noise=true , address=145.79.227.32:4836 , piece_id=DRKH7VQJJIBS23UBARHVQKHZ2XMAH24GVRDLPNXWBMOZSQAJEHDQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.584119254 (584h44m56.455202289s) [noise=true , address=145.79.227.46:4601 , piece_id=FKFRMZT7M3JS4QBY7GZVM5RFBZH26ZSEPWUWJAW4Q5PNIRQOZZJA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.585266789 (584h44m56.456349824s) [noise=true , address=145.79.227.6:4257 , piece_id=OINK3GXZ3LPJNOEGVQIIM4RQ47CLOB26OAC2ORRP2TBSMWVOMT7A, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.596427409 (584h44m56.467510454s) [noise=true , address=145.79.227.7:4295 , piece_id=645LDS7KTXFS6WBULVAJK4JF47LGQMMJARTBBA56SFWOX4JP34EA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.603525604 (584h44m56.47460864s) [noise=true , address=145.79.227.38:4978 , piece_id=T7QMNGNVNTJWXV7IFJVJYSJ4V254FABPDNQXP4OF2HINSJ4W37FA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.617699061 (584h44m56.488782106s) [noise=true , address=145.79.227.47:4266 , piece_id=FABBIGDGNGLNPUAKV3AWMUJWLBADI24JZU57YD3NVONNK6GJZOBQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.764432269 (584h44m56.635515314s) [noise=true , address=145.79.227.29:4941 , piece_id=ZFKGRHKZ7ZJ75E2S7PBT4NTFDEAIN5AGKFVDC6VDEW7WIGLDRE3Q, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.799518496 (584h44m56.670601541s) [noise=true , address=77.90.31.11:4554 , piece_id=F55TEVYP4YR6CSQCBXCDLZAAZIKHBFLHWIEAT4K4JJ3IDEXHVXPQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.831356326 (584h44m56.702439361s) [noise=true , address=45.91.116.6:4975 , piece_id=UI4XFR5ED5JU7RJMYJMH35CZPAM462AYTMAAK4M5YPCUJZASZV3A, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.881597158 (584h44m56.752680203s) [noise=true , address=45.91.116.12:4225 , piece_id=KMYYROA46XZ2QQLP77JLXL5M4BQZDYZUDFPFB36YUAJGPS3XJCLQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.881642113 (584h44m56.752725148s) [seg_pos={0 0}, upload=512597]: Segment reached optimal threshold of 13 pieces. Dec 30 00:13:56.881977061 (584h44m56.753060096s) [noise=true , address=77.90.31.19:4841 , piece_id=YW2UDCQC7H7GGHJ5JYITHI73INFNZC5Q4GXE7DYZNAZAO45QW73A, seg_pos={0 0}, upload=512597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 30 00:13:56.881998051 (584h44m56.753081096s) [noise=true , address=77.90.31.19:4841 , piece_id=YW2UDCQC7H7GGHJ5JYITHI73INFNZC5Q4GXE7DYZNAZAO45QW73A, seg_pos={0 0}, upload=512597]: Long tail cancellation due to optimal threshold Dec 30 00:13:56.882072983 (584h44m56.753156018s) [noise=true , address=45.91.116.13:4154 , piece_id=RDETXZUSPHCNDATGFT25HGLHJMAA54TU7Z5GTHGOAICPCLX5AJYA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 30 00:13:56.882081300 (584h44m56.753164335s) [noise=true , address=45.91.116.13:4154 , piece_id=RDETXZUSPHCNDATGFT25HGLHJMAA54TU7Z5GTHGOAICPCLX5AJYA, seg_pos={0 0}, upload=512597]: Long tail cancellation due to optimal threshold Dec 30 00:13:56.882140361 (584h44m56.753223396s) [noise=true , address=45.91.116.14:4189 , piece_id=RLVQQCO37ZG3RN742DKERZDNN6ALSQGTUZAFBW4SZUF2UOBSK6KA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:40250->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:40250->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40250->45.91.116.14:4189: use of closed network connection Dec 30 00:13:56.882149750 (584h44m56.753232785s) [noise=true , address=45.91.116.14:4189 , piece_id=RLVQQCO37ZG3RN742DKERZDNN6ALSQGTUZAFBW4SZUF2UOBSK6KA, seg_pos={0 0}, upload=512597]: Long tail cancellation due to optimal threshold Dec 30 00:13:56.882166521 (584h44m56.753249557s) [seg_pos={0 0}, upload=512597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:13:56.882257545 (584h44m56.75334058s) [upload=512597]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:13:56.882262575 (584h44m56.75334561s) [upload=512597]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:13:57.173987584 (584h44m57.045070629s) [upload=512597]: Done uploading segments... Dec 30 00:13:57.174005758 (584h44m57.045088793s) [upload=512597]: Upload finished. err: <nil> Duration: 6.310468613s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXX............XXX....X...X..........X......XX.X............] duration: 5.78782923s trace: https://jaeger.dev.storj.io/trace/1f21e11794748e52 FAIL (2025-12-30 00:23:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: KMYYROA46XZ2QQLP77JLXL5M4BQZDYZUDFPFB36YUAJGPS3XJCLQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: UI4XFR5ED5JU7RJMYJMH35CZPAM462AYTMAAK4M5YPCUJZASZV3A) context deadline exceeded ======LOGS====== 30 00:13:54.324867912 (584h44m54.195950957s) [upload=512597]: Starting upload Dec 30 00:13:54.324990224 (584h44m54.196073259s) [upload=512597]: Done starting upload Dec 30 00:13:54.325054557 (584h44m54.196137602s) [upload=512597]: Uploading segments... Dec 30 00:13:54.325419562 (584h44m54.196502607s) [upload=512597]: Got next segment. Inline: false Dec 30 00:13:54.325469307 (584h44m54.196552362s) [upload=512597]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:13:54.325475618 (584h44m54.196558664s) [upload=512597]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:13:54.580461825 (584h44m54.45154487s) [seg_pos={0 0}, upload=512597]: Begin upload segment... Dec 30 00:13:54.580507753 (584h44m54.451590798s) [seg_pos={0 0}, upload=512597]: Done begin upload segment. Dec 30 00:13:54.580544733 (584h44m54.451627768s) [noise=true , address=145.79.227.37:4567 , piece_id=QBSRM467II7QRO73BZMG5YCI5DGBJ3YJPKTGA4JTNFW7MVE4GCNA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.580601511 (584h44m54.451684556s) [noise=true , address=45.91.116.12:4225 , piece_id=KMYYROA46XZ2QQLP77JLXL5M4BQZDYZUDFPFB36YUAJGPS3XJCLQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.581263281 (584h44m54.452346327s) [noise=true , address=145.79.227.22:4509 , piece_id=IAFA5FPN2CO2EGSYPDH3ZNLMHOF4VIMDWNKPF4JMWQ7UDX7UGMKQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.581718839 (584h44m54.452801874s) [noise=true , address=145.79.227.32:4836 , piece_id=DRKH7VQJJIBS23UBARHVQKHZ2XMAH24GVRDLPNXWBMOZSQAJEHDQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.582118329 (584h44m54.453201364s) [noise=true , address=45.91.116.6:4975 , piece_id=UI4XFR5ED5JU7RJMYJMH35CZPAM462AYTMAAK4M5YPCUJZASZV3A, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.582437077 (584h44m54.453520112s) [noise=true , address=45.91.116.13:4154 , piece_id=RDETXZUSPHCNDATGFT25HGLHJMAA54TU7Z5GTHGOAICPCLX5AJYA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.582894558 (584h44m54.453977603s) [noise=true , address=77.90.31.11:4554 , piece_id=F55TEVYP4YR6CSQCBXCDLZAAZIKHBFLHWIEAT4K4JJ3IDEXHVXPQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.583261416 (584h44m54.454344461s) [noise=true , address=145.79.227.15:4874 , piece_id=H5NXNYBGQFR2IIIAY4T4IOEYFDWLVYQETIJWYJJ4HXI43GEESIDQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.583612384 (584h44m54.454695459s) [noise=true , address=145.79.227.29:4941 , piece_id=ZFKGRHKZ7ZJ75E2S7PBT4NTFDEAIN5AGKFVDC6VDEW7WIGLDRE3Q, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.585392814 (584h44m54.456475859s) [noise=true , address=145.79.227.6:4257 , piece_id=OINK3GXZ3LPJNOEGVQIIM4RQ47CLOB26OAC2ORRP2TBSMWVOMT7A, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.585658430 (584h44m54.456741465s) [noise=true , address=145.79.227.38:4978 , piece_id=T7QMNGNVNTJWXV7IFJVJYSJ4V254FABPDNQXP4OF2HINSJ4W37FA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.587224062 (584h44m54.458307097s) [noise=true , address=145.79.227.47:4266 , piece_id=FABBIGDGNGLNPUAKV3AWMUJWLBADI24JZU57YD3NVONNK6GJZOBQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.588454895 (584h44m54.45953793s) [noise=true , address=77.90.31.19:4841 , piece_id=YW2UDCQC7H7GGHJ5JYITHI73INFNZC5Q4GXE7DYZNAZAO45QW73A, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.589580087 (584h44m54.460663132s) [noise=true , address=45.91.116.14:4189 , piece_id=RLVQQCO37ZG3RN742DKERZDNN6ALSQGTUZAFBW4SZUF2UOBSK6KA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.591136290 (584h44m54.462219325s) [noise=true , address=145.79.227.46:4601 , piece_id=FKFRMZT7M3JS4QBY7GZVM5RFBZH26ZSEPWUWJAW4Q5PNIRQOZZJA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.592682314 (584h44m54.463765359s) [noise=true , address=145.79.227.7:4295 , piece_id=645LDS7KTXFS6WBULVAJK4JF47LGQMMJARTBBA56SFWOX4JP34EA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:56.281718857 (584h44m56.152801902s) [upload=512597]: Next returned nil segment Dec 30 00:13:56.281741130 (584h44m56.152824175s) [upload=512597]: Waiting for error group managing segments... Dec 30 00:13:56.563110793 (584h44m56.434193828s) [noise=true , address=145.79.227.15:4874 , piece_id=H5NXNYBGQFR2IIIAY4T4IOEYFDWLVYQETIJWYJJ4HXI43GEESIDQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.570998270 (584h44m56.442081315s) [noise=true , address=145.79.227.22:4509 , piece_id=IAFA5FPN2CO2EGSYPDH3ZNLMHOF4VIMDWNKPF4JMWQ7UDX7UGMKQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.572245676 (584h44m56.443328721s) [noise=true , address=145.79.227.37:4567 , piece_id=QBSRM467II7QRO73BZMG5YCI5DGBJ3YJPKTGA4JTNFW7MVE4GCNA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.579941789 (584h44m56.451024824s) [noise=true , address=145.79.227.32:4836 , piece_id=DRKH7VQJJIBS23UBARHVQKHZ2XMAH24GVRDLPNXWBMOZSQAJEHDQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.584119254 (584h44m56.455202289s) [noise=true , address=145.79.227.46:4601 , piece_id=FKFRMZT7M3JS4QBY7GZVM5RFBZH26ZSEPWUWJAW4Q5PNIRQOZZJA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.585266789 (584h44m56.456349824s) [noise=true , address=145.79.227.6:4257 , piece_id=OINK3GXZ3LPJNOEGVQIIM4RQ47CLOB26OAC2ORRP2TBSMWVOMT7A, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.596427409 (584h44m56.467510454s) [noise=true , address=145.79.227.7:4295 , piece_id=645LDS7KTXFS6WBULVAJK4JF47LGQMMJARTBBA56SFWOX4JP34EA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.603525604 (584h44m56.47460864s) [noise=true , address=145.79.227.38:4978 , piece_id=T7QMNGNVNTJWXV7IFJVJYSJ4V254FABPDNQXP4OF2HINSJ4W37FA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.617699061 (584h44m56.488782106s) [noise=true , address=145.79.227.47:4266 , piece_id=FABBIGDGNGLNPUAKV3AWMUJWLBADI24JZU57YD3NVONNK6GJZOBQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.764432269 (584h44m56.635515314s) [noise=true , address=145.79.227.29:4941 , piece_id=ZFKGRHKZ7ZJ75E2S7PBT4NTFDEAIN5AGKFVDC6VDEW7WIGLDRE3Q, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.799518496 (584h44m56.670601541s) [noise=true , address=77.90.31.11:4554 , piece_id=F55TEVYP4YR6CSQCBXCDLZAAZIKHBFLHWIEAT4K4JJ3IDEXHVXPQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.831356326 (584h44m56.702439361s) [noise=true , address=45.91.116.6:4975 , piece_id=UI4XFR5ED5JU7RJMYJMH35CZPAM462AYTMAAK4M5YPCUJZASZV3A, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.881597158 (584h44m56.752680203s) [noise=true , address=45.91.116.12:4225 , piece_id=KMYYROA46XZ2QQLP77JLXL5M4BQZDYZUDFPFB36YUAJGPS3XJCLQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.881642113 (584h44m56.752725148s) [seg_pos={0 0}, upload=512597]: Segment reached optimal threshold of 13 pieces. Dec 30 00:13:56.881977061 (584h44m56.753060096s) [noise=true , address=77.90.31.19:4841 , piece_id=YW2UDCQC7H7GGHJ5JYITHI73INFNZC5Q4GXE7DYZNAZAO45QW73A, seg_pos={0 0}, upload=512597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 30 00:13:56.881998051 (584h44m56.753081096s) [noise=true , address=77.90.31.19:4841 , piece_id=YW2UDCQC7H7GGHJ5JYITHI73INFNZC5Q4GXE7DYZNAZAO45QW73A, seg_pos={0 0}, upload=512597]: Long tail cancellation due to optimal threshold Dec 30 00:13:56.882072983 (584h44m56.753156018s) [noise=true , address=45.91.116.13:4154 , piece_id=RDETXZUSPHCNDATGFT25HGLHJMAA54TU7Z5GTHGOAICPCLX5AJYA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 30 00:13:56.882081300 (584h44m56.753164335s) [noise=true , address=45.91.116.13:4154 , piece_id=RDETXZUSPHCNDATGFT25HGLHJMAA54TU7Z5GTHGOAICPCLX5AJYA, seg_pos={0 0}, upload=512597]: Long tail cancellation due to optimal threshold Dec 30 00:13:56.882140361 (584h44m56.753223396s) [noise=true , address=45.91.116.14:4189 , piece_id=RLVQQCO37ZG3RN742DKERZDNN6ALSQGTUZAFBW4SZUF2UOBSK6KA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:40250->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:40250->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40250->45.91.116.14:4189: use of closed network connection Dec 30 00:13:56.882149750 (584h44m56.753232785s) [noise=true , address=45.91.116.14:4189 , piece_id=RLVQQCO37ZG3RN742DKERZDNN6ALSQGTUZAFBW4SZUF2UOBSK6KA, seg_pos={0 0}, upload=512597]: Long tail cancellation due to optimal threshold Dec 30 00:13:56.882166521 (584h44m56.753249557s) [seg_pos={0 0}, upload=512597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:13:56.882257545 (584h44m56.75334058s) [upload=512597]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:13:56.882262575 (584h44m56.75334561s) [upload=512597]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:13:57.173987584 (584h44m57.045070629s) [upload=512597]: Done uploading segments... Dec 30 00:13:57.174005758 (584h44m57.045088793s) [upload=512597]: Upload finished. err: <nil> Duration: 5.78785595s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.569781701s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:23:13 UTC) orbiter warning: context deadline exceeded Duration: 3.569810286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 36.581520894s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:23:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 36.58154024s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XXX............XXX....X...X..........X......XX.X...........] duration: 21.200553621s trace: https://jaeger.dev.storj.io/trace/1f21e11794748e52 FAIL (2025-12-30 00:23:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: KMYYROA46XZ2QQLP77JLXL5M4BQZDYZUDFPFB36YUAJGPS3XJCLQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: UI4XFR5ED5JU7RJMYJMH35CZPAM462AYTMAAK4M5YPCUJZASZV3A) context deadline exceeded ======LOGS====== 30 00:13:54.324867912 (584h44m54.195950957s) [upload=512597]: Starting upload Dec 30 00:13:54.324990224 (584h44m54.196073259s) [upload=512597]: Done starting upload Dec 30 00:13:54.325054557 (584h44m54.196137602s) [upload=512597]: Uploading segments... Dec 30 00:13:54.325419562 (584h44m54.196502607s) [upload=512597]: Got next segment. Inline: false Dec 30 00:13:54.325469307 (584h44m54.196552362s) [upload=512597]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 00:13:54.325475618 (584h44m54.196558664s) [upload=512597]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 00:13:54.580461825 (584h44m54.45154487s) [seg_pos={0 0}, upload=512597]: Begin upload segment... Dec 30 00:13:54.580507753 (584h44m54.451590798s) [seg_pos={0 0}, upload=512597]: Done begin upload segment. Dec 30 00:13:54.580544733 (584h44m54.451627768s) [noise=true , address=145.79.227.37:4567 , piece_id=QBSRM467II7QRO73BZMG5YCI5DGBJ3YJPKTGA4JTNFW7MVE4GCNA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.580601511 (584h44m54.451684556s) [noise=true , address=45.91.116.12:4225 , piece_id=KMYYROA46XZ2QQLP77JLXL5M4BQZDYZUDFPFB36YUAJGPS3XJCLQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.581263281 (584h44m54.452346327s) [noise=true , address=145.79.227.22:4509 , piece_id=IAFA5FPN2CO2EGSYPDH3ZNLMHOF4VIMDWNKPF4JMWQ7UDX7UGMKQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.581718839 (584h44m54.452801874s) [noise=true , address=145.79.227.32:4836 , piece_id=DRKH7VQJJIBS23UBARHVQKHZ2XMAH24GVRDLPNXWBMOZSQAJEHDQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.582118329 (584h44m54.453201364s) [noise=true , address=45.91.116.6:4975 , piece_id=UI4XFR5ED5JU7RJMYJMH35CZPAM462AYTMAAK4M5YPCUJZASZV3A, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.582437077 (584h44m54.453520112s) [noise=true , address=45.91.116.13:4154 , piece_id=RDETXZUSPHCNDATGFT25HGLHJMAA54TU7Z5GTHGOAICPCLX5AJYA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.582894558 (584h44m54.453977603s) [noise=true , address=77.90.31.11:4554 , piece_id=F55TEVYP4YR6CSQCBXCDLZAAZIKHBFLHWIEAT4K4JJ3IDEXHVXPQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.583261416 (584h44m54.454344461s) [noise=true , address=145.79.227.15:4874 , piece_id=H5NXNYBGQFR2IIIAY4T4IOEYFDWLVYQETIJWYJJ4HXI43GEESIDQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.583612384 (584h44m54.454695459s) [noise=true , address=145.79.227.29:4941 , piece_id=ZFKGRHKZ7ZJ75E2S7PBT4NTFDEAIN5AGKFVDC6VDEW7WIGLDRE3Q, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.585392814 (584h44m54.456475859s) [noise=true , address=145.79.227.6:4257 , piece_id=OINK3GXZ3LPJNOEGVQIIM4RQ47CLOB26OAC2ORRP2TBSMWVOMT7A, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.585658430 (584h44m54.456741465s) [noise=true , address=145.79.227.38:4978 , piece_id=T7QMNGNVNTJWXV7IFJVJYSJ4V254FABPDNQXP4OF2HINSJ4W37FA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.587224062 (584h44m54.458307097s) [noise=true , address=145.79.227.47:4266 , piece_id=FABBIGDGNGLNPUAKV3AWMUJWLBADI24JZU57YD3NVONNK6GJZOBQ, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.588454895 (584h44m54.45953793s) [noise=true , address=77.90.31.19:4841 , piece_id=YW2UDCQC7H7GGHJ5JYITHI73INFNZC5Q4GXE7DYZNAZAO45QW73A, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.589580087 (584h44m54.460663132s) [noise=true , address=45.91.116.14:4189 , piece_id=RLVQQCO37ZG3RN742DKERZDNN6ALSQGTUZAFBW4SZUF2UOBSK6KA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.591136290 (584h44m54.462219325s) [noise=true , address=145.79.227.46:4601 , piece_id=FKFRMZT7M3JS4QBY7GZVM5RFBZH26ZSEPWUWJAW4Q5PNIRQOZZJA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:54.592682314 (584h44m54.463765359s) [noise=true , address=145.79.227.7:4295 , piece_id=645LDS7KTXFS6WBULVAJK4JF47LGQMMJARTBBA56SFWOX4JP34EA, seg_pos={0 0}, upload=512597]: Uploading piece... Dec 30 00:13:56.281718857 (584h44m56.152801902s) [upload=512597]: Next returned nil segment Dec 30 00:13:56.281741130 (584h44m56.152824175s) [upload=512597]: Waiting for error group managing segments... Dec 30 00:13:56.563110793 (584h44m56.434193828s) [noise=true , address=145.79.227.15:4874 , piece_id=H5NXNYBGQFR2IIIAY4T4IOEYFDWLVYQETIJWYJJ4HXI43GEESIDQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.570998270 (584h44m56.442081315s) [noise=true , address=145.79.227.22:4509 , piece_id=IAFA5FPN2CO2EGSYPDH3ZNLMHOF4VIMDWNKPF4JMWQ7UDX7UGMKQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.572245676 (584h44m56.443328721s) [noise=true , address=145.79.227.37:4567 , piece_id=QBSRM467II7QRO73BZMG5YCI5DGBJ3YJPKTGA4JTNFW7MVE4GCNA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.579941789 (584h44m56.451024824s) [noise=true , address=145.79.227.32:4836 , piece_id=DRKH7VQJJIBS23UBARHVQKHZ2XMAH24GVRDLPNXWBMOZSQAJEHDQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.584119254 (584h44m56.455202289s) [noise=true , address=145.79.227.46:4601 , piece_id=FKFRMZT7M3JS4QBY7GZVM5RFBZH26ZSEPWUWJAW4Q5PNIRQOZZJA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.585266789 (584h44m56.456349824s) [noise=true , address=145.79.227.6:4257 , piece_id=OINK3GXZ3LPJNOEGVQIIM4RQ47CLOB26OAC2ORRP2TBSMWVOMT7A, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.596427409 (584h44m56.467510454s) [noise=true , address=145.79.227.7:4295 , piece_id=645LDS7KTXFS6WBULVAJK4JF47LGQMMJARTBBA56SFWOX4JP34EA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.603525604 (584h44m56.47460864s) [noise=true , address=145.79.227.38:4978 , piece_id=T7QMNGNVNTJWXV7IFJVJYSJ4V254FABPDNQXP4OF2HINSJ4W37FA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.617699061 (584h44m56.488782106s) [noise=true , address=145.79.227.47:4266 , piece_id=FABBIGDGNGLNPUAKV3AWMUJWLBADI24JZU57YD3NVONNK6GJZOBQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.764432269 (584h44m56.635515314s) [noise=true , address=145.79.227.29:4941 , piece_id=ZFKGRHKZ7ZJ75E2S7PBT4NTFDEAIN5AGKFVDC6VDEW7WIGLDRE3Q, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.799518496 (584h44m56.670601541s) [noise=true , address=77.90.31.11:4554 , piece_id=F55TEVYP4YR6CSQCBXCDLZAAZIKHBFLHWIEAT4K4JJ3IDEXHVXPQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.831356326 (584h44m56.702439361s) [noise=true , address=45.91.116.6:4975 , piece_id=UI4XFR5ED5JU7RJMYJMH35CZPAM462AYTMAAK4M5YPCUJZASZV3A, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.881597158 (584h44m56.752680203s) [noise=true , address=45.91.116.12:4225 , piece_id=KMYYROA46XZ2QQLP77JLXL5M4BQZDYZUDFPFB36YUAJGPS3XJCLQ, seg_pos={0 0}, upload=512597]: Done uploading piece. err: <nil> Dec 30 00:13:56.881642113 (584h44m56.752725148s) [seg_pos={0 0}, upload=512597]: Segment reached optimal threshold of 13 pieces. Dec 30 00:13:56.881977061 (584h44m56.753060096s) [noise=true , address=77.90.31.19:4841 , piece_id=YW2UDCQC7H7GGHJ5JYITHI73INFNZC5Q4GXE7DYZNAZAO45QW73A, seg_pos={0 0}, upload=512597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 30 00:13:56.881998051 (584h44m56.753081096s) [noise=true , address=77.90.31.19:4841 , piece_id=YW2UDCQC7H7GGHJ5JYITHI73INFNZC5Q4GXE7DYZNAZAO45QW73A, seg_pos={0 0}, upload=512597]: Long tail cancellation due to optimal threshold Dec 30 00:13:56.882072983 (584h44m56.753156018s) [noise=true , address=45.91.116.13:4154 , piece_id=RDETXZUSPHCNDATGFT25HGLHJMAA54TU7Z5GTHGOAICPCLX5AJYA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 30 00:13:56.882081300 (584h44m56.753164335s) [noise=true , address=45.91.116.13:4154 , piece_id=RDETXZUSPHCNDATGFT25HGLHJMAA54TU7Z5GTHGOAICPCLX5AJYA, seg_pos={0 0}, upload=512597]: Long tail cancellation due to optimal threshold Dec 30 00:13:56.882140361 (584h44m56.753223396s) [noise=true , address=45.91.116.14:4189 , piece_id=RLVQQCO37ZG3RN742DKERZDNN6ALSQGTUZAFBW4SZUF2UOBSK6KA, seg_pos={0 0}, upload=512597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:40250->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:40250->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40250->45.91.116.14:4189: use of closed network connection Dec 30 00:13:56.882149750 (584h44m56.753232785s) [noise=true , address=45.91.116.14:4189 , piece_id=RLVQQCO37ZG3RN742DKERZDNN6ALSQGTUZAFBW4SZUF2UOBSK6KA, seg_pos={0 0}, upload=512597]: Long tail cancellation due to optimal threshold Dec 30 00:13:56.882166521 (584h44m56.753249557s) [seg_pos={0 0}, upload=512597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 00:13:56.882257545 (584h44m56.75334058s) [upload=512597]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:13:56.882262575 (584h44m56.75334561s) [upload=512597]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:13:57.173987584 (584h44m57.045070629s) [upload=512597]: Done uploading segments... Dec 30 00:13:57.174005758 (584h44m57.045088793s) [upload=512597]: Upload finished. err: <nil> Duration: 21.20059531s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.48284003s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:22:49 UTC) orbiter warning: context deadline exceeded Duration: 3.482890686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 25.207064606s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:22:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 25.207091056s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.482241862s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:22:07 UTC) orbiter warning: context deadline exceeded Duration: 3.482272209s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)