Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 15.290486028s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:05:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 15.290516056s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.521546593s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:04:43 UTC) orbiter warning: context deadline exceeded Duration: 3.521572983s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 15.550080771s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:04:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 15.550097945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X......XX...X...X..XXX............XXX....X...X..........] duration: 6.157106054s trace: https://jaeger.dev.storj.io/trace/4cab5347a7a00b0e FAIL (2025-12-30 00:04:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ) context deadline exceeded ======LOGS====== Dec 29 23:56:42.167990647 (584h27m42.039073692s) [upload=512350]: Starting upload Dec 29 23:56:42.168086851 (584h27m42.039169896s) [upload=512350]: Done starting upload Dec 29 23:56:42.168131846 (584h27m42.039214911s) [upload=512350]: Uploading segments... Dec 29 23:56:42.168368076 (584h27m42.039451121s) [upload=512350]: Got next segment. Inline: false Dec 29 23:56:42.168428079 (584h27m42.039511135s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:56:42.168434873 (584h27m42.039517918s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:56:42.415701598 (584h27m42.286784633s) [seg_pos={0 0}, upload=512350]: Begin upload segment... Dec 29 23:56:42.415741133 (584h27m42.286824178s) [seg_pos={0 0}, upload=512350]: Done begin upload segment. Dec 29 23:56:42.415766301 (584h27m42.286849346s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.415774286 (584h27m42.286857321s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416481252 (584h27m42.287564297s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416841598 (584h27m42.287924633s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.417773663 (584h27m42.288856708s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418128278 (584h27m42.289211313s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418571592 (584h27m42.289654627s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.419739335 (584h27m42.29082237s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.420870890 (584h27m42.291953925s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.421748511 (584h27m42.292831546s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.423033568 (584h27m42.294116603s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.424234845 (584h27m42.29531788s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.425536333 (584h27m42.296619378s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.426180240 (584h27m42.297263285s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.427925353 (584h27m42.299008388s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.429464052 (584h27m42.300547087s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:44.115528639 (584h27m43.986611674s) [upload=512350]: Next returned nil segment Dec 29 23:56:44.115542315 (584h27m43.98662535s) [upload=512350]: Waiting for error group managing segments... Dec 29 23:56:44.397046451 (584h27m44.268129496s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.403792988 (584h27m44.274876023s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404426263 (584h27m44.275509308s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404531554 (584h27m44.275614589s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.405188705 (584h27m44.27627174s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.410954133 (584h27m44.282037168s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.412750262 (584h27m44.283833298s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.421747013 (584h27m44.292830048s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.423328385 (584h27m44.29441143s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.432806963 (584h27m44.303889998s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.591246478 (584h27m44.462329524s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.691932356 (584h27m44.563015391s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701663075 (584h27m44.57274611s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701692701 (584h27m44.572775736s) [seg_pos={0 0}, upload=512350]: Segment reached optimal threshold of 13 pieces. Dec 29 23:56:44.701970681 (584h27m44.573053726s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.701979848 (584h27m44.573062883s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702161814 (584h27m44.573244849s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702177363 (584h27m44.573260398s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702303744 (584h27m44.573386789s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702314394 (584h27m44.573397439s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702343310 (584h27m44.573426355s) [seg_pos={0 0}, upload=512350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:56:44.702546907 (584h27m44.573629953s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:56:44.702555383 (584h27m44.573638418s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:56:45.004134192 (584h27m44.875217257s) [upload=512350]: Done uploading segments... Dec 29 23:56:45.004153519 (584h27m44.875236554s) [upload=512350]: Upload finished. err: <nil> Duration: 6.157133697s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.003570537s trace: https://jaeger.dev.storj.io/trace/d9b5e7a0764073a FAIL (2025-12-30 00:04:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.054995084 (584h34m23.926078129s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055681120 (584h34m23.926764155s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.055727699 (584h34m23.926810734s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056172756 (584h34m23.927255801s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056325287 (584h34m23.927408332s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056586374 (584h34m23.927669409s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.056848083 (584h34m23.927931128s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057044406 (584h34m23.928127451s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057355188 (584h34m23.928438233s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057581158 (584h34m23.928664203s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.057909483 (584h34m23.928992518s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058132118 (584h34m23.929215153s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.058537499 (584h34m23.929620544s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059010821 (584h34m23.930093856s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059380223 (584h34m23.930463258s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059473051 (584h34m23.930556086s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:24.059908529 (584h34m23.930991574s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Uploading piece... Dec 30 00:03:25.509135800 (584h34m25.380218835s) [upload=512446]: Next returned nil segment Dec 30 00:03:25.509159946 (584h34m25.380242981s) [upload=512446]: Waiting for error group managing segments... Dec 30 00:03:25.933309894 (584h34m25.804392929s) [noise=true , address=145.79.227.2:4142 , piece_id=FXD24JNQJJP2VB67KEIEUXM3JVE7PZIMVPWAPY75HIHWTHLOJO5Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.936893258 (584h34m25.807976293s) [noise=true , address=145.79.227.28:4729 , piece_id=H2YI3WRSPM4SYLQYECEQB2P3BL5WQRUT2AD73MMHHIAEPIQ5OQKQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:25.964377010 (584h34m25.835460055s) [noise=true , address=5.231.73.5:4028 , piece_id=RRSISOHKK7ZCIUYTDP2DIDWBG7EGXR53HN375AM47W2VJCKI3IHA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.127305133 (584h34m25.998388168s) [noise=true , address=77.90.31.13:4646 , piece_id=EWU65MLWZN2HYUAIBJUCPCJJFPVPUMD2CLIZJCW6LB4RGQIQNH4A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.149601938 (584h34m26.020684973s) [noise=true , address=45.91.116.10:4089 , piece_id=QESSZBQM77B2KSFYXB5I4FTZRLUKOQZ2QS772XBTSOSYGIOLNSXA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.171327607 (584h34m26.042410652s) [noise=true , address=45.91.116.5:4324 , piece_id=7HRIGNDR6CTMYZ32GLYQRJNOCCGSNLA6K3CQQ5IDSS7RCDYJOHLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.178626855 (584h34m26.0497099s) [noise=true , address=77.90.31.14:4683 , piece_id=E6FHPWYXES77MO4DZMSJVEZALYV4CVXSHE3NSTV6V6PLVPH3SNNA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307072510 (584h34m26.178155555s) [noise=true , address=77.90.31.11:4563 , piece_id=HLT2VJLWW3DDZBSB33PPK26D4SOKYMI2SAPGUO6WZKVYB7D4FM6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.307452614 (584h34m26.178535669s) [noise=true , address=5.231.73.3:4007 , piece_id=57G2WTG4HKBF3RWNYBP4EX4UYLFPIGDQ7DKPYCT52P4DDRRBEKAQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.774555811 (584h34m26.645638856s) [noise=true , address=77.90.31.10:4525 , piece_id=ABRBS2KNST5JOTPALQEF7CWC572QBACCEA4SZJOJ44NPASDH7APQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.807074060 (584h34m26.678157105s) [noise=true , address=145.79.227.43:4172 , piece_id=ZLLEFKLZB4BTXAPPRXROWLJMIJH6BGLXKB3JDXEJKZ7WHCQEMAAA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.844249776 (584h34m26.715332811s) [noise=true , address=145.79.227.35:4491 , piece_id=HADQ5ZNJMTY4XUHH4DQIN4A5M7ZUNGWKA7WS5TGNEBQEEEP4RT6A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.857331805 (584h34m26.728414841s) [noise=true , address=145.79.227.44:4208 , piece_id=Y4CF6ZGVNXZAEPUXWRZQCIGWQQLNV5NOGVYFZD4FQ6UW6NKCN75A, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885916324 (584h34m26.756999369s) [noise=true , address=145.79.227.34:4916 , piece_id=65CJD4K6D7XPLGUGOJZ5QQHJQAR7CUC5G6ZYUB6M74PM5ZT4DTUA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: <nil> Dec 30 00:03:26.885935540 (584h34m26.757018575s) [seg_pos={0 0}, upload=512446]: Segment reached optimal threshold of 14 pieces. Dec 30 00:03:26.886114872 (584h34m26.757197907s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886121715 (584h34m26.75720475s) [noise=true , address=145.79.227.33:4875 , piece_id=HF5D4ZOZOBIUHD6EAIJVUS4VR5NAUDZXYINNYUQ3EK2W3DERCFSA, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886276640 (584h34m26.757359675s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886282661 (584h34m26.757365696s) [noise=true , address=145.79.227.49:4367 , piece_id=3PDVDN6GAZCXLZS2SUSV5BPVDRHJSYWSPXLHKEGSJCVIU2KHUA3Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886380808 (584h34m26.757463843s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886385557 (584h34m26.757468592s) [noise=true , address=145.79.227.16:4937 , piece_id=UTZPGXRZHMMUA7XR3YLSV6YV5EOBILWHYHH7MGDUBJJEI36VYNLQ, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886847857 (584h34m26.757930902s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7): protocol: expected piece hash; context canceled; EOF Dec 30 00:03:26.886976082 (584h34m26.758059117s) [noise=true , address=145.79.227.30:4776 , piece_id=WV3F6KLOZJ6CIIZWXOHHKI3SSKA6BM6GAOOGUXH7R2FKUQ4VX52Q, seg_pos={0 0}, upload=512446]: Long tail cancellation due to optimal threshold Dec 30 00:03:26.886992954 (584h34m26.758075989s) [seg_pos={0 0}, upload=512446]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 00:03:26.887054801 (584h34m26.758137846s) [upload=512446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 00:03:26.887059079 (584h34m26.758142114s) [upload=512446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 00:03:27.181348310 (584h34m27.052431345s) [upload=512446]: Done uploading segments... Dec 30 00:03:27.181368157 (584h34m27.052451192s) [upload=512446]: Upload finished. err: <nil> Duration: 1m0.003599915s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X......XX...X...X..XXX............XXX....X...X.........] duration: 5.723928538s trace: https://jaeger.dev.storj.io/trace/4cab5347a7a00b0e FAIL (2025-12-30 00:03:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ) context deadline exceeded ======LOGS====== Dec 29 23:56:42.167990647 (584h27m42.039073692s) [upload=512350]: Starting upload Dec 29 23:56:42.168086851 (584h27m42.039169896s) [upload=512350]: Done starting upload Dec 29 23:56:42.168131846 (584h27m42.039214911s) [upload=512350]: Uploading segments... Dec 29 23:56:42.168368076 (584h27m42.039451121s) [upload=512350]: Got next segment. Inline: false Dec 29 23:56:42.168428079 (584h27m42.039511135s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:56:42.168434873 (584h27m42.039517918s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:56:42.415701598 (584h27m42.286784633s) [seg_pos={0 0}, upload=512350]: Begin upload segment... Dec 29 23:56:42.415741133 (584h27m42.286824178s) [seg_pos={0 0}, upload=512350]: Done begin upload segment. Dec 29 23:56:42.415766301 (584h27m42.286849346s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.415774286 (584h27m42.286857321s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416481252 (584h27m42.287564297s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416841598 (584h27m42.287924633s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.417773663 (584h27m42.288856708s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418128278 (584h27m42.289211313s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418571592 (584h27m42.289654627s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.419739335 (584h27m42.29082237s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.420870890 (584h27m42.291953925s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.421748511 (584h27m42.292831546s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.423033568 (584h27m42.294116603s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.424234845 (584h27m42.29531788s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.425536333 (584h27m42.296619378s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.426180240 (584h27m42.297263285s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.427925353 (584h27m42.299008388s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.429464052 (584h27m42.300547087s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:44.115528639 (584h27m43.986611674s) [upload=512350]: Next returned nil segment Dec 29 23:56:44.115542315 (584h27m43.98662535s) [upload=512350]: Waiting for error group managing segments... Dec 29 23:56:44.397046451 (584h27m44.268129496s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.403792988 (584h27m44.274876023s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404426263 (584h27m44.275509308s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404531554 (584h27m44.275614589s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.405188705 (584h27m44.27627174s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.410954133 (584h27m44.282037168s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.412750262 (584h27m44.283833298s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.421747013 (584h27m44.292830048s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.423328385 (584h27m44.29441143s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.432806963 (584h27m44.303889998s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.591246478 (584h27m44.462329524s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.691932356 (584h27m44.563015391s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701663075 (584h27m44.57274611s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701692701 (584h27m44.572775736s) [seg_pos={0 0}, upload=512350]: Segment reached optimal threshold of 13 pieces. Dec 29 23:56:44.701970681 (584h27m44.573053726s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.701979848 (584h27m44.573062883s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702161814 (584h27m44.573244849s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702177363 (584h27m44.573260398s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702303744 (584h27m44.573386789s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702314394 (584h27m44.573397439s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702343310 (584h27m44.573426355s) [seg_pos={0 0}, upload=512350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:56:44.702546907 (584h27m44.573629953s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:56:44.702555383 (584h27m44.573638418s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:56:45.004134192 (584h27m44.875217257s) [upload=512350]: Done uploading segments... Dec 29 23:56:45.004153519 (584h27m44.875236554s) [upload=512350]: Upload finished. err: <nil> Duration: 5.723952784s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.392771237s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:03:51 UTC) orbiter warning: context deadline exceeded Duration: 3.392785784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.....X......XX...X...X..XXX............XXX....X...X........] duration: 10.711602469s trace: https://jaeger.dev.storj.io/trace/4cab5347a7a00b0e FAIL (2025-12-30 00:03:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ) context deadline exceeded ======LOGS====== Dec 29 23:56:42.167990647 (584h27m42.039073692s) [upload=512350]: Starting upload Dec 29 23:56:42.168086851 (584h27m42.039169896s) [upload=512350]: Done starting upload Dec 29 23:56:42.168131846 (584h27m42.039214911s) [upload=512350]: Uploading segments... Dec 29 23:56:42.168368076 (584h27m42.039451121s) [upload=512350]: Got next segment. Inline: false Dec 29 23:56:42.168428079 (584h27m42.039511135s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:56:42.168434873 (584h27m42.039517918s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:56:42.415701598 (584h27m42.286784633s) [seg_pos={0 0}, upload=512350]: Begin upload segment... Dec 29 23:56:42.415741133 (584h27m42.286824178s) [seg_pos={0 0}, upload=512350]: Done begin upload segment. Dec 29 23:56:42.415766301 (584h27m42.286849346s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.415774286 (584h27m42.286857321s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416481252 (584h27m42.287564297s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416841598 (584h27m42.287924633s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.417773663 (584h27m42.288856708s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418128278 (584h27m42.289211313s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418571592 (584h27m42.289654627s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.419739335 (584h27m42.29082237s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.420870890 (584h27m42.291953925s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.421748511 (584h27m42.292831546s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.423033568 (584h27m42.294116603s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.424234845 (584h27m42.29531788s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.425536333 (584h27m42.296619378s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.426180240 (584h27m42.297263285s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.427925353 (584h27m42.299008388s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.429464052 (584h27m42.300547087s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:44.115528639 (584h27m43.986611674s) [upload=512350]: Next returned nil segment Dec 29 23:56:44.115542315 (584h27m43.98662535s) [upload=512350]: Waiting for error group managing segments... Dec 29 23:56:44.397046451 (584h27m44.268129496s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.403792988 (584h27m44.274876023s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404426263 (584h27m44.275509308s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404531554 (584h27m44.275614589s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.405188705 (584h27m44.27627174s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.410954133 (584h27m44.282037168s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.412750262 (584h27m44.283833298s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.421747013 (584h27m44.292830048s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.423328385 (584h27m44.29441143s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.432806963 (584h27m44.303889998s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.591246478 (584h27m44.462329524s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.691932356 (584h27m44.563015391s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701663075 (584h27m44.57274611s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701692701 (584h27m44.572775736s) [seg_pos={0 0}, upload=512350]: Segment reached optimal threshold of 13 pieces. Dec 29 23:56:44.701970681 (584h27m44.573053726s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.701979848 (584h27m44.573062883s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702161814 (584h27m44.573244849s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702177363 (584h27m44.573260398s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702303744 (584h27m44.573386789s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702314394 (584h27m44.573397439s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702343310 (584h27m44.573426355s) [seg_pos={0 0}, upload=512350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:56:44.702546907 (584h27m44.573629953s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:56:44.702555383 (584h27m44.573638418s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:56:45.004134192 (584h27m44.875217257s) [upload=512350]: Done uploading segments... Dec 29 23:56:45.004153519 (584h27m44.875236554s) [upload=512350]: Upload finished. err: <nil> Duration: 10.71162967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.....X......XX...X...X..XXX............XXX....X...X.......] duration: 12.559326863s trace: https://jaeger.dev.storj.io/trace/4cab5347a7a00b0e FAIL (2025-12-30 00:03:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ) context deadline exceeded ======LOGS====== Dec 29 23:56:42.167990647 (584h27m42.039073692s) [upload=512350]: Starting upload Dec 29 23:56:42.168086851 (584h27m42.039169896s) [upload=512350]: Done starting upload Dec 29 23:56:42.168131846 (584h27m42.039214911s) [upload=512350]: Uploading segments... Dec 29 23:56:42.168368076 (584h27m42.039451121s) [upload=512350]: Got next segment. Inline: false Dec 29 23:56:42.168428079 (584h27m42.039511135s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:56:42.168434873 (584h27m42.039517918s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:56:42.415701598 (584h27m42.286784633s) [seg_pos={0 0}, upload=512350]: Begin upload segment... Dec 29 23:56:42.415741133 (584h27m42.286824178s) [seg_pos={0 0}, upload=512350]: Done begin upload segment. Dec 29 23:56:42.415766301 (584h27m42.286849346s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.415774286 (584h27m42.286857321s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416481252 (584h27m42.287564297s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416841598 (584h27m42.287924633s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.417773663 (584h27m42.288856708s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418128278 (584h27m42.289211313s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418571592 (584h27m42.289654627s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.419739335 (584h27m42.29082237s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.420870890 (584h27m42.291953925s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.421748511 (584h27m42.292831546s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.423033568 (584h27m42.294116603s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.424234845 (584h27m42.29531788s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.425536333 (584h27m42.296619378s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.426180240 (584h27m42.297263285s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.427925353 (584h27m42.299008388s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.429464052 (584h27m42.300547087s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:44.115528639 (584h27m43.986611674s) [upload=512350]: Next returned nil segment Dec 29 23:56:44.115542315 (584h27m43.98662535s) [upload=512350]: Waiting for error group managing segments... Dec 29 23:56:44.397046451 (584h27m44.268129496s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.403792988 (584h27m44.274876023s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404426263 (584h27m44.275509308s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404531554 (584h27m44.275614589s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.405188705 (584h27m44.27627174s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.410954133 (584h27m44.282037168s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.412750262 (584h27m44.283833298s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.421747013 (584h27m44.292830048s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.423328385 (584h27m44.29441143s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.432806963 (584h27m44.303889998s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.591246478 (584h27m44.462329524s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.691932356 (584h27m44.563015391s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701663075 (584h27m44.57274611s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701692701 (584h27m44.572775736s) [seg_pos={0 0}, upload=512350]: Segment reached optimal threshold of 13 pieces. Dec 29 23:56:44.701970681 (584h27m44.573053726s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.701979848 (584h27m44.573062883s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702161814 (584h27m44.573244849s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702177363 (584h27m44.573260398s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702303744 (584h27m44.573386789s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702314394 (584h27m44.573397439s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702343310 (584h27m44.573426355s) [seg_pos={0 0}, upload=512350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:56:44.702546907 (584h27m44.573629953s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:56:44.702555383 (584h27m44.573638418s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:56:45.004134192 (584h27m44.875217257s) [upload=512350]: Done uploading segments... Dec 29 23:56:45.004153519 (584h27m44.875236554s) [upload=512350]: Upload finished. err: <nil> Duration: 12.559351179s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.496104223s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:02:43 UTC) orbiter warning: context deadline exceeded Duration: 3.496131575s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X.....X......XX...X...X..XXX............XXX....X...X......] duration: 7.800626888s trace: https://jaeger.dev.storj.io/trace/4cab5347a7a00b0e FAIL (2025-12-30 00:02:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ) context deadline exceeded ======LOGS====== Dec 29 23:56:42.167990647 (584h27m42.039073692s) [upload=512350]: Starting upload Dec 29 23:56:42.168086851 (584h27m42.039169896s) [upload=512350]: Done starting upload Dec 29 23:56:42.168131846 (584h27m42.039214911s) [upload=512350]: Uploading segments... Dec 29 23:56:42.168368076 (584h27m42.039451121s) [upload=512350]: Got next segment. Inline: false Dec 29 23:56:42.168428079 (584h27m42.039511135s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:56:42.168434873 (584h27m42.039517918s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:56:42.415701598 (584h27m42.286784633s) [seg_pos={0 0}, upload=512350]: Begin upload segment... Dec 29 23:56:42.415741133 (584h27m42.286824178s) [seg_pos={0 0}, upload=512350]: Done begin upload segment. Dec 29 23:56:42.415766301 (584h27m42.286849346s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.415774286 (584h27m42.286857321s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416481252 (584h27m42.287564297s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416841598 (584h27m42.287924633s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.417773663 (584h27m42.288856708s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418128278 (584h27m42.289211313s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418571592 (584h27m42.289654627s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.419739335 (584h27m42.29082237s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.420870890 (584h27m42.291953925s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.421748511 (584h27m42.292831546s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.423033568 (584h27m42.294116603s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.424234845 (584h27m42.29531788s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.425536333 (584h27m42.296619378s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.426180240 (584h27m42.297263285s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.427925353 (584h27m42.299008388s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.429464052 (584h27m42.300547087s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:44.115528639 (584h27m43.986611674s) [upload=512350]: Next returned nil segment Dec 29 23:56:44.115542315 (584h27m43.98662535s) [upload=512350]: Waiting for error group managing segments... Dec 29 23:56:44.397046451 (584h27m44.268129496s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.403792988 (584h27m44.274876023s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404426263 (584h27m44.275509308s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404531554 (584h27m44.275614589s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.405188705 (584h27m44.27627174s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.410954133 (584h27m44.282037168s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.412750262 (584h27m44.283833298s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.421747013 (584h27m44.292830048s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.423328385 (584h27m44.29441143s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.432806963 (584h27m44.303889998s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.591246478 (584h27m44.462329524s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.691932356 (584h27m44.563015391s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701663075 (584h27m44.57274611s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701692701 (584h27m44.572775736s) [seg_pos={0 0}, upload=512350]: Segment reached optimal threshold of 13 pieces. Dec 29 23:56:44.701970681 (584h27m44.573053726s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.701979848 (584h27m44.573062883s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702161814 (584h27m44.573244849s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702177363 (584h27m44.573260398s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702303744 (584h27m44.573386789s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702314394 (584h27m44.573397439s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702343310 (584h27m44.573426355s) [seg_pos={0 0}, upload=512350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:56:44.702546907 (584h27m44.573629953s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:56:44.702555383 (584h27m44.573638418s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:56:45.004134192 (584h27m44.875217257s) [upload=512350]: Done uploading segments... Dec 29 23:56:45.004153519 (584h27m44.875236554s) [upload=512350]: Upload finished. err: <nil> Duration: 7.800653239s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.428401277s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:01:32 UTC) orbiter warning: context deadline exceeded Duration: 3.428445181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X.....X......XX...X...X..XXX............XXX....X...X.....] duration: 5.876087126s trace: https://jaeger.dev.storj.io/trace/4cab5347a7a00b0e FAIL (2025-12-30 00:01:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ) context deadline exceeded ======LOGS====== Dec 29 23:56:42.167990647 (584h27m42.039073692s) [upload=512350]: Starting upload Dec 29 23:56:42.168086851 (584h27m42.039169896s) [upload=512350]: Done starting upload Dec 29 23:56:42.168131846 (584h27m42.039214911s) [upload=512350]: Uploading segments... Dec 29 23:56:42.168368076 (584h27m42.039451121s) [upload=512350]: Got next segment. Inline: false Dec 29 23:56:42.168428079 (584h27m42.039511135s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:56:42.168434873 (584h27m42.039517918s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:56:42.415701598 (584h27m42.286784633s) [seg_pos={0 0}, upload=512350]: Begin upload segment... Dec 29 23:56:42.415741133 (584h27m42.286824178s) [seg_pos={0 0}, upload=512350]: Done begin upload segment. Dec 29 23:56:42.415766301 (584h27m42.286849346s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.415774286 (584h27m42.286857321s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416481252 (584h27m42.287564297s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416841598 (584h27m42.287924633s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.417773663 (584h27m42.288856708s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418128278 (584h27m42.289211313s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418571592 (584h27m42.289654627s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.419739335 (584h27m42.29082237s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.420870890 (584h27m42.291953925s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.421748511 (584h27m42.292831546s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.423033568 (584h27m42.294116603s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.424234845 (584h27m42.29531788s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.425536333 (584h27m42.296619378s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.426180240 (584h27m42.297263285s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.427925353 (584h27m42.299008388s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.429464052 (584h27m42.300547087s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:44.115528639 (584h27m43.986611674s) [upload=512350]: Next returned nil segment Dec 29 23:56:44.115542315 (584h27m43.98662535s) [upload=512350]: Waiting for error group managing segments... Dec 29 23:56:44.397046451 (584h27m44.268129496s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.403792988 (584h27m44.274876023s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404426263 (584h27m44.275509308s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404531554 (584h27m44.275614589s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.405188705 (584h27m44.27627174s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.410954133 (584h27m44.282037168s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.412750262 (584h27m44.283833298s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.421747013 (584h27m44.292830048s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.423328385 (584h27m44.29441143s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.432806963 (584h27m44.303889998s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.591246478 (584h27m44.462329524s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.691932356 (584h27m44.563015391s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701663075 (584h27m44.57274611s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701692701 (584h27m44.572775736s) [seg_pos={0 0}, upload=512350]: Segment reached optimal threshold of 13 pieces. Dec 29 23:56:44.701970681 (584h27m44.573053726s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.701979848 (584h27m44.573062883s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702161814 (584h27m44.573244849s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702177363 (584h27m44.573260398s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702303744 (584h27m44.573386789s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702314394 (584h27m44.573397439s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702343310 (584h27m44.573426355s) [seg_pos={0 0}, upload=512350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:56:44.702546907 (584h27m44.573629953s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:56:44.702555383 (584h27m44.573638418s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:56:45.004134192 (584h27m44.875217257s) [upload=512350]: Done uploading segments... Dec 29 23:56:45.004153519 (584h27m44.875236554s) [upload=512350]: Upload finished. err: <nil> Duration: 5.876117224s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.512539556s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-30 00:00:41 UTC) orbiter warning: context deadline exceeded Duration: 3.512574042s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....X.....X......XX...X...X..XXX............XXX....X...X....] duration: 7.79403386s trace: https://jaeger.dev.storj.io/trace/4cab5347a7a00b0e FAIL (2025-12-30 00:00:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ) context deadline exceeded ======LOGS====== Dec 29 23:56:42.167990647 (584h27m42.039073692s) [upload=512350]: Starting upload Dec 29 23:56:42.168086851 (584h27m42.039169896s) [upload=512350]: Done starting upload Dec 29 23:56:42.168131846 (584h27m42.039214911s) [upload=512350]: Uploading segments... Dec 29 23:56:42.168368076 (584h27m42.039451121s) [upload=512350]: Got next segment. Inline: false Dec 29 23:56:42.168428079 (584h27m42.039511135s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:56:42.168434873 (584h27m42.039517918s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:56:42.415701598 (584h27m42.286784633s) [seg_pos={0 0}, upload=512350]: Begin upload segment... Dec 29 23:56:42.415741133 (584h27m42.286824178s) [seg_pos={0 0}, upload=512350]: Done begin upload segment. Dec 29 23:56:42.415766301 (584h27m42.286849346s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.415774286 (584h27m42.286857321s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416481252 (584h27m42.287564297s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416841598 (584h27m42.287924633s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.417773663 (584h27m42.288856708s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418128278 (584h27m42.289211313s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418571592 (584h27m42.289654627s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.419739335 (584h27m42.29082237s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.420870890 (584h27m42.291953925s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.421748511 (584h27m42.292831546s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.423033568 (584h27m42.294116603s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.424234845 (584h27m42.29531788s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.425536333 (584h27m42.296619378s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.426180240 (584h27m42.297263285s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.427925353 (584h27m42.299008388s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.429464052 (584h27m42.300547087s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:44.115528639 (584h27m43.986611674s) [upload=512350]: Next returned nil segment Dec 29 23:56:44.115542315 (584h27m43.98662535s) [upload=512350]: Waiting for error group managing segments... Dec 29 23:56:44.397046451 (584h27m44.268129496s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.403792988 (584h27m44.274876023s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404426263 (584h27m44.275509308s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404531554 (584h27m44.275614589s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.405188705 (584h27m44.27627174s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.410954133 (584h27m44.282037168s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.412750262 (584h27m44.283833298s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.421747013 (584h27m44.292830048s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.423328385 (584h27m44.29441143s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.432806963 (584h27m44.303889998s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.591246478 (584h27m44.462329524s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.691932356 (584h27m44.563015391s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701663075 (584h27m44.57274611s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701692701 (584h27m44.572775736s) [seg_pos={0 0}, upload=512350]: Segment reached optimal threshold of 13 pieces. Dec 29 23:56:44.701970681 (584h27m44.573053726s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.701979848 (584h27m44.573062883s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702161814 (584h27m44.573244849s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702177363 (584h27m44.573260398s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702303744 (584h27m44.573386789s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702314394 (584h27m44.573397439s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702343310 (584h27m44.573426355s) [seg_pos={0 0}, upload=512350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:56:44.702546907 (584h27m44.573629953s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:56:44.702555383 (584h27m44.573638418s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:56:45.004134192 (584h27m44.875217257s) [upload=512350]: Done uploading segments... Dec 29 23:56:45.004153519 (584h27m44.875236554s) [upload=512350]: Upload finished. err: <nil> Duration: 7.794099976s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.553497845s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-29 23:59:51 UTC) orbiter warning: context deadline exceeded Duration: 3.553526089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X....X.....X......XX...X...X..XXX............XXX....X...X...] duration: 6.084873492s trace: https://jaeger.dev.storj.io/trace/4cab5347a7a00b0e FAIL (2025-12-29 23:59:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ) context deadline exceeded ======LOGS====== Dec 29 23:56:42.167990647 (584h27m42.039073692s) [upload=512350]: Starting upload Dec 29 23:56:42.168086851 (584h27m42.039169896s) [upload=512350]: Done starting upload Dec 29 23:56:42.168131846 (584h27m42.039214911s) [upload=512350]: Uploading segments... Dec 29 23:56:42.168368076 (584h27m42.039451121s) [upload=512350]: Got next segment. Inline: false Dec 29 23:56:42.168428079 (584h27m42.039511135s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:56:42.168434873 (584h27m42.039517918s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:56:42.415701598 (584h27m42.286784633s) [seg_pos={0 0}, upload=512350]: Begin upload segment... Dec 29 23:56:42.415741133 (584h27m42.286824178s) [seg_pos={0 0}, upload=512350]: Done begin upload segment. Dec 29 23:56:42.415766301 (584h27m42.286849346s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.415774286 (584h27m42.286857321s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416481252 (584h27m42.287564297s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416841598 (584h27m42.287924633s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.417773663 (584h27m42.288856708s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418128278 (584h27m42.289211313s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418571592 (584h27m42.289654627s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.419739335 (584h27m42.29082237s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.420870890 (584h27m42.291953925s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.421748511 (584h27m42.292831546s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.423033568 (584h27m42.294116603s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.424234845 (584h27m42.29531788s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.425536333 (584h27m42.296619378s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.426180240 (584h27m42.297263285s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.427925353 (584h27m42.299008388s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.429464052 (584h27m42.300547087s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:44.115528639 (584h27m43.986611674s) [upload=512350]: Next returned nil segment Dec 29 23:56:44.115542315 (584h27m43.98662535s) [upload=512350]: Waiting for error group managing segments... Dec 29 23:56:44.397046451 (584h27m44.268129496s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.403792988 (584h27m44.274876023s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404426263 (584h27m44.275509308s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404531554 (584h27m44.275614589s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.405188705 (584h27m44.27627174s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.410954133 (584h27m44.282037168s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.412750262 (584h27m44.283833298s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.421747013 (584h27m44.292830048s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.423328385 (584h27m44.29441143s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.432806963 (584h27m44.303889998s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.591246478 (584h27m44.462329524s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.691932356 (584h27m44.563015391s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701663075 (584h27m44.57274611s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701692701 (584h27m44.572775736s) [seg_pos={0 0}, upload=512350]: Segment reached optimal threshold of 13 pieces. Dec 29 23:56:44.701970681 (584h27m44.573053726s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.701979848 (584h27m44.573062883s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702161814 (584h27m44.573244849s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702177363 (584h27m44.573260398s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702303744 (584h27m44.573386789s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702314394 (584h27m44.573397439s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702343310 (584h27m44.573426355s) [seg_pos={0 0}, upload=512350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:56:44.702546907 (584h27m44.573629953s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:56:44.702555383 (584h27m44.573638418s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:56:45.004134192 (584h27m44.875217257s) [upload=512350]: Done uploading segments... Dec 29 23:56:45.004153519 (584h27m44.875236554s) [upload=512350]: Upload finished. err: <nil> Duration: 6.084899201s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.49646607s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-29 23:59:23 UTC) orbiter warning: context deadline exceeded Duration: 3.496491256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X....X.....X......XX...X...X..XXX............XXX....X...X..] duration: 6.313705807s trace: https://jaeger.dev.storj.io/trace/4cab5347a7a00b0e FAIL (2025-12-29 23:58:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ) context deadline exceeded ======LOGS====== Dec 29 23:56:42.167990647 (584h27m42.039073692s) [upload=512350]: Starting upload Dec 29 23:56:42.168086851 (584h27m42.039169896s) [upload=512350]: Done starting upload Dec 29 23:56:42.168131846 (584h27m42.039214911s) [upload=512350]: Uploading segments... Dec 29 23:56:42.168368076 (584h27m42.039451121s) [upload=512350]: Got next segment. Inline: false Dec 29 23:56:42.168428079 (584h27m42.039511135s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:56:42.168434873 (584h27m42.039517918s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:56:42.415701598 (584h27m42.286784633s) [seg_pos={0 0}, upload=512350]: Begin upload segment... Dec 29 23:56:42.415741133 (584h27m42.286824178s) [seg_pos={0 0}, upload=512350]: Done begin upload segment. Dec 29 23:56:42.415766301 (584h27m42.286849346s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.415774286 (584h27m42.286857321s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416481252 (584h27m42.287564297s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416841598 (584h27m42.287924633s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.417773663 (584h27m42.288856708s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418128278 (584h27m42.289211313s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418571592 (584h27m42.289654627s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.419739335 (584h27m42.29082237s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.420870890 (584h27m42.291953925s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.421748511 (584h27m42.292831546s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.423033568 (584h27m42.294116603s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.424234845 (584h27m42.29531788s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.425536333 (584h27m42.296619378s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.426180240 (584h27m42.297263285s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.427925353 (584h27m42.299008388s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.429464052 (584h27m42.300547087s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:44.115528639 (584h27m43.986611674s) [upload=512350]: Next returned nil segment Dec 29 23:56:44.115542315 (584h27m43.98662535s) [upload=512350]: Waiting for error group managing segments... Dec 29 23:56:44.397046451 (584h27m44.268129496s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.403792988 (584h27m44.274876023s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404426263 (584h27m44.275509308s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404531554 (584h27m44.275614589s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.405188705 (584h27m44.27627174s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.410954133 (584h27m44.282037168s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.412750262 (584h27m44.283833298s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.421747013 (584h27m44.292830048s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.423328385 (584h27m44.29441143s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.432806963 (584h27m44.303889998s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.591246478 (584h27m44.462329524s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.691932356 (584h27m44.563015391s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701663075 (584h27m44.57274611s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701692701 (584h27m44.572775736s) [seg_pos={0 0}, upload=512350]: Segment reached optimal threshold of 13 pieces. Dec 29 23:56:44.701970681 (584h27m44.573053726s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.701979848 (584h27m44.573062883s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702161814 (584h27m44.573244849s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702177363 (584h27m44.573260398s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702303744 (584h27m44.573386789s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702314394 (584h27m44.573397439s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702343310 (584h27m44.573426355s) [seg_pos={0 0}, upload=512350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:56:44.702546907 (584h27m44.573629953s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:56:44.702555383 (584h27m44.573638418s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:56:45.004134192 (584h27m44.875217257s) [upload=512350]: Done uploading segments... Dec 29 23:56:45.004153519 (584h27m44.875236554s) [upload=512350]: Upload finished. err: <nil> Duration: 6.313745452s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.548903577s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-29 23:58:21 UTC) orbiter warning: context deadline exceeded Duration: 3.548972638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X....X.....X......XX...X...X..XXX............XXX....X...X.] duration: 6.237528949s trace: https://jaeger.dev.storj.io/trace/4cab5347a7a00b0e FAIL (2025-12-29 23:57:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ) context deadline exceeded ======LOGS====== Dec 29 23:56:42.167990647 (584h27m42.039073692s) [upload=512350]: Starting upload Dec 29 23:56:42.168086851 (584h27m42.039169896s) [upload=512350]: Done starting upload Dec 29 23:56:42.168131846 (584h27m42.039214911s) [upload=512350]: Uploading segments... Dec 29 23:56:42.168368076 (584h27m42.039451121s) [upload=512350]: Got next segment. Inline: false Dec 29 23:56:42.168428079 (584h27m42.039511135s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:56:42.168434873 (584h27m42.039517918s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:56:42.415701598 (584h27m42.286784633s) [seg_pos={0 0}, upload=512350]: Begin upload segment... Dec 29 23:56:42.415741133 (584h27m42.286824178s) [seg_pos={0 0}, upload=512350]: Done begin upload segment. Dec 29 23:56:42.415766301 (584h27m42.286849346s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.415774286 (584h27m42.286857321s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416481252 (584h27m42.287564297s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416841598 (584h27m42.287924633s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.417773663 (584h27m42.288856708s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418128278 (584h27m42.289211313s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418571592 (584h27m42.289654627s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.419739335 (584h27m42.29082237s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.420870890 (584h27m42.291953925s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.421748511 (584h27m42.292831546s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.423033568 (584h27m42.294116603s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.424234845 (584h27m42.29531788s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.425536333 (584h27m42.296619378s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.426180240 (584h27m42.297263285s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.427925353 (584h27m42.299008388s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.429464052 (584h27m42.300547087s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:44.115528639 (584h27m43.986611674s) [upload=512350]: Next returned nil segment Dec 29 23:56:44.115542315 (584h27m43.98662535s) [upload=512350]: Waiting for error group managing segments... Dec 29 23:56:44.397046451 (584h27m44.268129496s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.403792988 (584h27m44.274876023s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404426263 (584h27m44.275509308s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404531554 (584h27m44.275614589s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.405188705 (584h27m44.27627174s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.410954133 (584h27m44.282037168s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.412750262 (584h27m44.283833298s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.421747013 (584h27m44.292830048s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.423328385 (584h27m44.29441143s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.432806963 (584h27m44.303889998s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.591246478 (584h27m44.462329524s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.691932356 (584h27m44.563015391s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701663075 (584h27m44.57274611s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701692701 (584h27m44.572775736s) [seg_pos={0 0}, upload=512350]: Segment reached optimal threshold of 13 pieces. Dec 29 23:56:44.701970681 (584h27m44.573053726s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.701979848 (584h27m44.573062883s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702161814 (584h27m44.573244849s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702177363 (584h27m44.573260398s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702303744 (584h27m44.573386789s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702314394 (584h27m44.573397439s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702343310 (584h27m44.573426355s) [seg_pos={0 0}, upload=512350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:56:44.702546907 (584h27m44.573629953s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:56:44.702555383 (584h27m44.573638418s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:56:45.004134192 (584h27m44.875217257s) [upload=512350]: Done uploading segments... Dec 29 23:56:45.004153519 (584h27m44.875236554s) [upload=512350]: Upload finished. err: <nil> Duration: 6.237555389s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.554336273s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-29 23:57:18 UTC) orbiter warning: context deadline exceeded Duration: 3.554359246s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..X....X.....X......XX...X...X..XXX............XXX....X...X] duration: 30.00221608s trace: https://jaeger.dev.storj.io/trace/4cab5347a7a00b0e FAIL (2025-12-29 23:57:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ) context deadline exceeded ======LOGS====== Dec 29 23:56:42.167990647 (584h27m42.039073692s) [upload=512350]: Starting upload Dec 29 23:56:42.168086851 (584h27m42.039169896s) [upload=512350]: Done starting upload Dec 29 23:56:42.168131846 (584h27m42.039214911s) [upload=512350]: Uploading segments... Dec 29 23:56:42.168368076 (584h27m42.039451121s) [upload=512350]: Got next segment. Inline: false Dec 29 23:56:42.168428079 (584h27m42.039511135s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:56:42.168434873 (584h27m42.039517918s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:56:42.415701598 (584h27m42.286784633s) [seg_pos={0 0}, upload=512350]: Begin upload segment... Dec 29 23:56:42.415741133 (584h27m42.286824178s) [seg_pos={0 0}, upload=512350]: Done begin upload segment. Dec 29 23:56:42.415766301 (584h27m42.286849346s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.415774286 (584h27m42.286857321s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416481252 (584h27m42.287564297s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.416841598 (584h27m42.287924633s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.417773663 (584h27m42.288856708s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418128278 (584h27m42.289211313s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.418571592 (584h27m42.289654627s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.419739335 (584h27m42.29082237s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.420870890 (584h27m42.291953925s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.421748511 (584h27m42.292831546s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.423033568 (584h27m42.294116603s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.424234845 (584h27m42.29531788s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.425536333 (584h27m42.296619378s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.426180240 (584h27m42.297263285s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.427925353 (584h27m42.299008388s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:42.429464052 (584h27m42.300547087s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Uploading piece... Dec 29 23:56:44.115528639 (584h27m43.986611674s) [upload=512350]: Next returned nil segment Dec 29 23:56:44.115542315 (584h27m43.98662535s) [upload=512350]: Waiting for error group managing segments... Dec 29 23:56:44.397046451 (584h27m44.268129496s) [noise=true , address=145.79.227.51:4410 , piece_id=WQJORQ77WBM7PDVAHOA7XNLF6AYYOET26BK737MOWMIDQOJIQXYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.403792988 (584h27m44.274876023s) [noise=true , address=145.79.227.10:4402 , piece_id=AR3WBWJ6735N6V7LKNWLZFYTQMXZD7SARHTB6VN3H5PLTLH762MA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404426263 (584h27m44.275509308s) [noise=true , address=145.79.227.11:4730 , piece_id=VKXQQWTBIDJYNOZOWUX6LQB2XYHMAY32UDLMBFIJ2FLUTVSNQMYA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.404531554 (584h27m44.275614589s) [noise=true , address=145.79.227.12:4765 , piece_id=TGUNJ53P5O6L3PCEQLQ67UFESOD57BTUUPCFS4XWAJW3RNUVYBMQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.405188705 (584h27m44.27627174s) [noise=true , address=145.79.227.18:4007 , piece_id=UPAQDJZ6DHKBORL43SVX4MQ3UCKJ3ZBUWPUEU6ZW3LCP6AQM42TQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.410954133 (584h27m44.282037168s) [noise=true , address=145.79.227.42:4122 , piece_id=D5EW6OUV7UWTNMHSLR5CB5LDBEFYLFP5ZHOAH5PQPEJLFWDM6BKA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.412750262 (584h27m44.283833298s) [noise=true , address=145.79.227.7:4295 , piece_id=NO3L7AB6FBGYBYBHL6FU4HT3EPERDTF6F6NK73K6WBU3ELJM4MMA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.421747013 (584h27m44.292830048s) [noise=true , address=145.79.227.14:4839 , piece_id=3OLDQBD5VAYJX3ZLQX3JMFOM3FFRZBVGONP2A2C5LVPPM4R3MMHQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.423328385 (584h27m44.29441143s) [noise=true , address=145.79.227.6:4258 , piece_id=U3B2AVNOFBPDIGZT36KB2L6YZR5UULQ7L7AZJO2QKHCDZDZSOOEA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.432806963 (584h27m44.303889998s) [noise=true , address=145.79.227.45:4231 , piece_id=ANOXUK2G3D2ULROIFGY3KWUVVHFY7SI2CQ2KJRYT7FWYX3ASLRNA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.591246478 (584h27m44.462329524s) [noise=true , address=77.90.31.16:4735 , piece_id=7NOPHSGDOYZ37XVRFTVON2NY542ANI7PE6DVLKEXECRBPWTQXPFQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.691932356 (584h27m44.563015391s) [noise=true , address=45.91.116.12:4227 , piece_id=DYXAPEEWWNVZX6MR2MMPW5IN6CPIIQOCAWKKZVUNE6J6IRPNLKVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701663075 (584h27m44.57274611s) [noise=true , address=45.91.116.10:4084 , piece_id=5LQBAR3DGMD7FXODYDFKHKXGQEXUO4IKNHHJKM6OBBC7BS62ITAQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: <nil> Dec 29 23:56:44.701692701 (584h27m44.572775736s) [seg_pos={0 0}, upload=512350]: Segment reached optimal threshold of 13 pieces. Dec 29 23:56:44.701970681 (584h27m44.573053726s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.701979848 (584h27m44.573062883s) [noise=true , address=77.90.31.18:4808 , piece_id=DD6U3MPQFM6VQQOTIRDRYLFXJNZK4WY3ZC5N7SXBLBAOU6A55AVQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702161814 (584h27m44.573244849s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702177363 (584h27m44.573260398s) [noise=true , address=45.91.116.13:4155 , piece_id=7J6EUV76WGIGNGSAC3OZVXXI36UB2MTDEYLDR3YEV6N2Q25Y4DJA, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702303744 (584h27m44.573386789s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 29 23:56:44.702314394 (584h27m44.573397439s) [noise=true , address=77.90.31.21:4878 , piece_id=HB5WYBUNR6FKGGAXS7UKDFG2YGSMY4AVLSOAHHTMC4WYA36EAHEQ, seg_pos={0 0}, upload=512350]: Long tail cancellation due to optimal threshold Dec 29 23:56:44.702343310 (584h27m44.573426355s) [seg_pos={0 0}, upload=512350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:56:44.702546907 (584h27m44.573629953s) [upload=512350]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:56:44.702555383 (584h27m44.573638418s) [upload=512350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:56:45.004134192 (584h27m44.875217257s) [upload=512350]: Done uploading segments... Dec 29 23:56:45.004153519 (584h27m44.875236554s) [upload=512350]: Upload finished. err: <nil> Duration: 30.002241438s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.540918565s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-29 23:56:35 UTC) orbiter warning: context deadline exceeded Duration: 3.540945675s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X..X....X.....X......XX...X...X..XXX............XXX....X...] duration: 27.154410872s trace: https://jaeger.dev.storj.io/trace/3ed94279973ea3fd FAIL (2025-12-29 23:56:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 23:52:40.396609305 (584h23m40.26769234s) [upload=512289]: Starting upload Dec 29 23:52:40.396695880 (584h23m40.267778915s) [upload=512289]: Done starting upload Dec 29 23:52:40.396802342 (584h23m40.267885377s) [upload=512289]: Uploading segments... Dec 29 23:52:40.397087595 (584h23m40.26817064s) [upload=512289]: Got next segment. Inline: false Dec 29 23:52:40.397160774 (584h23m40.26824381s) [upload=512289]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:52:40.397168290 (584h23m40.268251335s) [upload=512289]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:52:40.662507012 (584h23m40.533590047s) [seg_pos={0 0}, upload=512289]: Begin upload segment... Dec 29 23:52:40.662584950 (584h23m40.533667995s) [seg_pos={0 0}, upload=512289]: Done begin upload segment. Dec 29 23:52:40.662613565 (584h23m40.53369661s) [noise=true , address=145.79.227.34:4905 , piece_id=XAPWWVZFZOYUB4SIXZFVUNRW7MN3TELYNSVF7YBHR2LHGH7QIUBQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.663209670 (584h23m40.534292715s) [noise=true , address=45.91.116.14:4192 , piece_id=6R3COZT6RRLVBXZY6D6ZMYBVKDCFIJ2XC4LSHFUY4W437IRWSGYQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.663881980 (584h23m40.534965015s) [noise=true , address=145.79.227.14:4837 , piece_id=6OAWQS7BFFLL6GIFUJ6HYCYUK7XATDVGEIJD64EZIHMM4HMZFYHQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.664173766 (584h23m40.535256801s) [noise=true , address=145.79.227.33:4871 , piece_id=K4Q4N5JDEWF2BOM3MB2QYPJ37FB3VZOLDZESXPJGK4SVVTIVLU6A, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.664464128 (584h23m40.535547174s) [noise=true , address=145.79.227.39:4013 , piece_id=64DIJVBDVC577RN63PMQ2CJVYAIMWBFA4G7MKQXPAFQUL2LTPVHQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.664794538 (584h23m40.535877573s) [noise=true , address=145.79.227.12:4765 , piece_id=UGDO5GNOL7ZNOYOJUTBUYWFAJ5VA34TOTRBHXGETSOILOKF6BQGA, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.665126139 (584h23m40.536209184s) [noise=true , address=145.79.227.3:4150 , piece_id=EGR7H3B3P3P5UJD22HN2SFVC5QRMSJPH32QVMUK4RZPUCTTTQ7CA, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.666091417 (584h23m40.537174452s) [noise=true , address=145.79.227.18:4006 , piece_id=FFKXKMKFC2LWZSF24Z23KYPIXXJNUEWZ5BDOCH5BC6RCQWHBUZDQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.666243066 (584h23m40.537326101s) [noise=true , address=45.91.116.11:4117 , piece_id=MUMUQKWEIY5XD7HNPP5CQWU226V3CMCYV4QWBI3FJO3C5HW7TE2Q, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.667229253 (584h23m40.538312298s) [noise=true , address=45.91.116.10:4082 , piece_id=PLERXO5CDHLCODU6A4NT2I347PGFFOLWLW6OJCRZTOLZDVUDZSKA, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.667347840 (584h23m40.538430875s) [noise=true , address=145.79.227.16:4910 , piece_id=DB5I6XG6DCMDELUVBO4J4MDPMFWOSVZVBKHLTBIRKHMXKFN5UW6A, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.668268052 (584h23m40.539351097s) [noise=true , address=145.79.227.46:4602 , piece_id=JJGLE5CLZBNRVLHVATHFMP54AM3XQALOCYUGBWFRCDBEUXOIKGMQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.669317530 (584h23m40.540400565s) [noise=true , address=77.90.31.14:4663 , piece_id=FQPF4ELL2S5GUSGKJPACQ437KHO25ULWBQDVYWTU6D6KYKTI2R7Q, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.669704568 (584h23m40.540787603s) [noise=true , address=45.91.116.8:4047 , piece_id=UIYYVKGDUJOP5I54WB6C4ILHKWTPDPXF33UCWZO2VMTHUJVZZ7RA, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.671770611 (584h23m40.542853646s) [noise=true , address=145.79.227.17:4972 , piece_id=YKOXP4ICO3YS36XJBP3XDJHAXLFR7TPWAIQUWB7VLZPUIMLMBPWQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.672991927 (584h23m40.544074962s) [noise=true , address=145.79.227.38:4978 , piece_id=QCOLYC3U5BNCMQYZHQYFO2P37TXE5WHTVD6JC47N2NY65YI6FDBQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:42.272017614 (584h23m42.143100659s) [upload=512289]: Next returned nil segment Dec 29 23:52:42.272029707 (584h23m42.143112742s) [upload=512289]: Waiting for error group managing segments... Dec 29 23:52:42.653571787 (584h23m42.524654822s) [noise=true , address=145.79.227.18:4006 , piece_id=FFKXKMKFC2LWZSF24Z23KYPIXXJNUEWZ5BDOCH5BC6RCQWHBUZDQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.654325893 (584h23m42.525408928s) [noise=true , address=145.79.227.14:4837 , piece_id=6OAWQS7BFFLL6GIFUJ6HYCYUK7XATDVGEIJD64EZIHMM4HMZFYHQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.656840582 (584h23m42.527923627s) [noise=true , address=145.79.227.12:4765 , piece_id=UGDO5GNOL7ZNOYOJUTBUYWFAJ5VA34TOTRBHXGETSOILOKF6BQGA, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.673972653 (584h23m42.545055698s) [noise=true , address=145.79.227.3:4150 , piece_id=EGR7H3B3P3P5UJD22HN2SFVC5QRMSJPH32QVMUK4RZPUCTTTQ7CA, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.679012039 (584h23m42.550095074s) [noise=true , address=145.79.227.39:4013 , piece_id=64DIJVBDVC577RN63PMQ2CJVYAIMWBFA4G7MKQXPAFQUL2LTPVHQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.679304616 (584h23m42.550387651s) [noise=true , address=145.79.227.16:4910 , piece_id=DB5I6XG6DCMDELUVBO4J4MDPMFWOSVZVBKHLTBIRKHMXKFN5UW6A, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.680776659 (584h23m42.551859704s) [noise=true , address=145.79.227.38:4978 , piece_id=QCOLYC3U5BNCMQYZHQYFO2P37TXE5WHTVD6JC47N2NY65YI6FDBQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.682216040 (584h23m42.553299075s) [noise=true , address=145.79.227.17:4972 , piece_id=YKOXP4ICO3YS36XJBP3XDJHAXLFR7TPWAIQUWB7VLZPUIMLMBPWQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.843092541 (584h23m42.714175586s) [noise=true , address=77.90.31.14:4663 , piece_id=FQPF4ELL2S5GUSGKJPACQ437KHO25ULWBQDVYWTU6D6KYKTI2R7Q, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.913519134 (584h23m42.784602179s) [noise=true , address=45.91.116.10:4082 , piece_id=PLERXO5CDHLCODU6A4NT2I347PGFFOLWLW6OJCRZTOLZDVUDZSKA, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.916411311 (584h23m42.787494376s) [noise=true , address=45.91.116.11:4117 , piece_id=MUMUQKWEIY5XD7HNPP5CQWU226V3CMCYV4QWBI3FJO3C5HW7TE2Q, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.963330993 (584h23m42.834414028s) [noise=true , address=45.91.116.8:4047 , piece_id=UIYYVKGDUJOP5I54WB6C4ILHKWTPDPXF33UCWZO2VMTHUJVZZ7RA, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:43.009589565 (584h23m42.880672611s) [noise=true , address=45.91.116.14:4192 , piece_id=6R3COZT6RRLVBXZY6D6ZMYBVKDCFIJ2XC4LSHFUY4W437IRWSGYQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:43.009610375 (584h23m42.880693421s) [seg_pos={0 0}, upload=512289]: Segment reached optimal threshold of 13 pieces. Dec 29 23:52:43.009841385 (584h23m42.88092442s) [noise=true , address=145.79.227.33:4871 , piece_id=K4Q4N5JDEWF2BOM3MB2QYPJ37FB3VZOLDZESXPJGK4SVVTIVLU6A, seg_pos={0 0}, upload=512289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 29 23:52:43.009848288 (584h23m42.880931333s) [noise=true , address=145.79.227.33:4871 , piece_id=K4Q4N5JDEWF2BOM3MB2QYPJ37FB3VZOLDZESXPJGK4SVVTIVLU6A, seg_pos={0 0}, upload=512289]: Long tail cancellation due to optimal threshold Dec 29 23:52:43.009995278 (584h23m42.881078323s) [noise=true , address=145.79.227.34:4905 , piece_id=XAPWWVZFZOYUB4SIXZFVUNRW7MN3TELYNSVF7YBHR2LHGH7QIUBQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 29 23:52:43.010010898 (584h23m42.881093943s) [noise=true , address=145.79.227.34:4905 , piece_id=XAPWWVZFZOYUB4SIXZFVUNRW7MN3TELYNSVF7YBHR2LHGH7QIUBQ, seg_pos={0 0}, upload=512289]: Long tail cancellation due to optimal threshold Dec 29 23:52:43.010089697 (584h23m42.881172742s) [noise=true , address=145.79.227.46:4602 , piece_id=JJGLE5CLZBNRVLHVATHFMP54AM3XQALOCYUGBWFRCDBEUXOIKGMQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d): protocol: expected piece hash; context canceled; EOF Dec 29 23:52:43.010097262 (584h23m42.881180297s) [noise=true , address=145.79.227.46:4602 , piece_id=JJGLE5CLZBNRVLHVATHFMP54AM3XQALOCYUGBWFRCDBEUXOIKGMQ, seg_pos={0 0}, upload=512289]: Long tail cancellation due to optimal threshold Dec 29 23:52:43.010117701 (584h23m42.881200736s) [seg_pos={0 0}, upload=512289]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:52:43.010192613 (584h23m42.881275648s) [upload=512289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:52:43.010198064 (584h23m42.881281099s) [upload=512289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:52:43.302225465 (584h23m43.1733085s) [upload=512289]: Done uploading segments... Dec 29 23:52:43.302243670 (584h23m43.173326705s) [upload=512289]: Upload finished. err: <nil> Duration: 27.154428976s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.680829838s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-29 23:55:41 UTC) orbiter warning: context deadline exceeded Duration: 3.680876397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X..X....X.....X......XX...X...X..XXX............XXX....X..] duration: 8.035998347s trace: https://jaeger.dev.storj.io/trace/3ed94279973ea3fd FAIL (2025-12-29 23:55:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 23:52:40.396609305 (584h23m40.26769234s) [upload=512289]: Starting upload Dec 29 23:52:40.396695880 (584h23m40.267778915s) [upload=512289]: Done starting upload Dec 29 23:52:40.396802342 (584h23m40.267885377s) [upload=512289]: Uploading segments... Dec 29 23:52:40.397087595 (584h23m40.26817064s) [upload=512289]: Got next segment. Inline: false Dec 29 23:52:40.397160774 (584h23m40.26824381s) [upload=512289]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:52:40.397168290 (584h23m40.268251335s) [upload=512289]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:52:40.662507012 (584h23m40.533590047s) [seg_pos={0 0}, upload=512289]: Begin upload segment... Dec 29 23:52:40.662584950 (584h23m40.533667995s) [seg_pos={0 0}, upload=512289]: Done begin upload segment. Dec 29 23:52:40.662613565 (584h23m40.53369661s) [noise=true , address=145.79.227.34:4905 , piece_id=XAPWWVZFZOYUB4SIXZFVUNRW7MN3TELYNSVF7YBHR2LHGH7QIUBQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.663209670 (584h23m40.534292715s) [noise=true , address=45.91.116.14:4192 , piece_id=6R3COZT6RRLVBXZY6D6ZMYBVKDCFIJ2XC4LSHFUY4W437IRWSGYQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.663881980 (584h23m40.534965015s) [noise=true , address=145.79.227.14:4837 , piece_id=6OAWQS7BFFLL6GIFUJ6HYCYUK7XATDVGEIJD64EZIHMM4HMZFYHQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.664173766 (584h23m40.535256801s) [noise=true , address=145.79.227.33:4871 , piece_id=K4Q4N5JDEWF2BOM3MB2QYPJ37FB3VZOLDZESXPJGK4SVVTIVLU6A, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.664464128 (584h23m40.535547174s) [noise=true , address=145.79.227.39:4013 , piece_id=64DIJVBDVC577RN63PMQ2CJVYAIMWBFA4G7MKQXPAFQUL2LTPVHQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.664794538 (584h23m40.535877573s) [noise=true , address=145.79.227.12:4765 , piece_id=UGDO5GNOL7ZNOYOJUTBUYWFAJ5VA34TOTRBHXGETSOILOKF6BQGA, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.665126139 (584h23m40.536209184s) [noise=true , address=145.79.227.3:4150 , piece_id=EGR7H3B3P3P5UJD22HN2SFVC5QRMSJPH32QVMUK4RZPUCTTTQ7CA, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.666091417 (584h23m40.537174452s) [noise=true , address=145.79.227.18:4006 , piece_id=FFKXKMKFC2LWZSF24Z23KYPIXXJNUEWZ5BDOCH5BC6RCQWHBUZDQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.666243066 (584h23m40.537326101s) [noise=true , address=45.91.116.11:4117 , piece_id=MUMUQKWEIY5XD7HNPP5CQWU226V3CMCYV4QWBI3FJO3C5HW7TE2Q, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.667229253 (584h23m40.538312298s) [noise=true , address=45.91.116.10:4082 , piece_id=PLERXO5CDHLCODU6A4NT2I347PGFFOLWLW6OJCRZTOLZDVUDZSKA, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.667347840 (584h23m40.538430875s) [noise=true , address=145.79.227.16:4910 , piece_id=DB5I6XG6DCMDELUVBO4J4MDPMFWOSVZVBKHLTBIRKHMXKFN5UW6A, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.668268052 (584h23m40.539351097s) [noise=true , address=145.79.227.46:4602 , piece_id=JJGLE5CLZBNRVLHVATHFMP54AM3XQALOCYUGBWFRCDBEUXOIKGMQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.669317530 (584h23m40.540400565s) [noise=true , address=77.90.31.14:4663 , piece_id=FQPF4ELL2S5GUSGKJPACQ437KHO25ULWBQDVYWTU6D6KYKTI2R7Q, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.669704568 (584h23m40.540787603s) [noise=true , address=45.91.116.8:4047 , piece_id=UIYYVKGDUJOP5I54WB6C4ILHKWTPDPXF33UCWZO2VMTHUJVZZ7RA, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.671770611 (584h23m40.542853646s) [noise=true , address=145.79.227.17:4972 , piece_id=YKOXP4ICO3YS36XJBP3XDJHAXLFR7TPWAIQUWB7VLZPUIMLMBPWQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.672991927 (584h23m40.544074962s) [noise=true , address=145.79.227.38:4978 , piece_id=QCOLYC3U5BNCMQYZHQYFO2P37TXE5WHTVD6JC47N2NY65YI6FDBQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:42.272017614 (584h23m42.143100659s) [upload=512289]: Next returned nil segment Dec 29 23:52:42.272029707 (584h23m42.143112742s) [upload=512289]: Waiting for error group managing segments... Dec 29 23:52:42.653571787 (584h23m42.524654822s) [noise=true , address=145.79.227.18:4006 , piece_id=FFKXKMKFC2LWZSF24Z23KYPIXXJNUEWZ5BDOCH5BC6RCQWHBUZDQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.654325893 (584h23m42.525408928s) [noise=true , address=145.79.227.14:4837 , piece_id=6OAWQS7BFFLL6GIFUJ6HYCYUK7XATDVGEIJD64EZIHMM4HMZFYHQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.656840582 (584h23m42.527923627s) [noise=true , address=145.79.227.12:4765 , piece_id=UGDO5GNOL7ZNOYOJUTBUYWFAJ5VA34TOTRBHXGETSOILOKF6BQGA, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.673972653 (584h23m42.545055698s) [noise=true , address=145.79.227.3:4150 , piece_id=EGR7H3B3P3P5UJD22HN2SFVC5QRMSJPH32QVMUK4RZPUCTTTQ7CA, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.679012039 (584h23m42.550095074s) [noise=true , address=145.79.227.39:4013 , piece_id=64DIJVBDVC577RN63PMQ2CJVYAIMWBFA4G7MKQXPAFQUL2LTPVHQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.679304616 (584h23m42.550387651s) [noise=true , address=145.79.227.16:4910 , piece_id=DB5I6XG6DCMDELUVBO4J4MDPMFWOSVZVBKHLTBIRKHMXKFN5UW6A, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.680776659 (584h23m42.551859704s) [noise=true , address=145.79.227.38:4978 , piece_id=QCOLYC3U5BNCMQYZHQYFO2P37TXE5WHTVD6JC47N2NY65YI6FDBQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.682216040 (584h23m42.553299075s) [noise=true , address=145.79.227.17:4972 , piece_id=YKOXP4ICO3YS36XJBP3XDJHAXLFR7TPWAIQUWB7VLZPUIMLMBPWQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.843092541 (584h23m42.714175586s) [noise=true , address=77.90.31.14:4663 , piece_id=FQPF4ELL2S5GUSGKJPACQ437KHO25ULWBQDVYWTU6D6KYKTI2R7Q, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.913519134 (584h23m42.784602179s) [noise=true , address=45.91.116.10:4082 , piece_id=PLERXO5CDHLCODU6A4NT2I347PGFFOLWLW6OJCRZTOLZDVUDZSKA, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.916411311 (584h23m42.787494376s) [noise=true , address=45.91.116.11:4117 , piece_id=MUMUQKWEIY5XD7HNPP5CQWU226V3CMCYV4QWBI3FJO3C5HW7TE2Q, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.963330993 (584h23m42.834414028s) [noise=true , address=45.91.116.8:4047 , piece_id=UIYYVKGDUJOP5I54WB6C4ILHKWTPDPXF33UCWZO2VMTHUJVZZ7RA, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:43.009589565 (584h23m42.880672611s) [noise=true , address=45.91.116.14:4192 , piece_id=6R3COZT6RRLVBXZY6D6ZMYBVKDCFIJ2XC4LSHFUY4W437IRWSGYQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:43.009610375 (584h23m42.880693421s) [seg_pos={0 0}, upload=512289]: Segment reached optimal threshold of 13 pieces. Dec 29 23:52:43.009841385 (584h23m42.88092442s) [noise=true , address=145.79.227.33:4871 , piece_id=K4Q4N5JDEWF2BOM3MB2QYPJ37FB3VZOLDZESXPJGK4SVVTIVLU6A, seg_pos={0 0}, upload=512289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 29 23:52:43.009848288 (584h23m42.880931333s) [noise=true , address=145.79.227.33:4871 , piece_id=K4Q4N5JDEWF2BOM3MB2QYPJ37FB3VZOLDZESXPJGK4SVVTIVLU6A, seg_pos={0 0}, upload=512289]: Long tail cancellation due to optimal threshold Dec 29 23:52:43.009995278 (584h23m42.881078323s) [noise=true , address=145.79.227.34:4905 , piece_id=XAPWWVZFZOYUB4SIXZFVUNRW7MN3TELYNSVF7YBHR2LHGH7QIUBQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 29 23:52:43.010010898 (584h23m42.881093943s) [noise=true , address=145.79.227.34:4905 , piece_id=XAPWWVZFZOYUB4SIXZFVUNRW7MN3TELYNSVF7YBHR2LHGH7QIUBQ, seg_pos={0 0}, upload=512289]: Long tail cancellation due to optimal threshold Dec 29 23:52:43.010089697 (584h23m42.881172742s) [noise=true , address=145.79.227.46:4602 , piece_id=JJGLE5CLZBNRVLHVATHFMP54AM3XQALOCYUGBWFRCDBEUXOIKGMQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d): protocol: expected piece hash; context canceled; EOF Dec 29 23:52:43.010097262 (584h23m42.881180297s) [noise=true , address=145.79.227.46:4602 , piece_id=JJGLE5CLZBNRVLHVATHFMP54AM3XQALOCYUGBWFRCDBEUXOIKGMQ, seg_pos={0 0}, upload=512289]: Long tail cancellation due to optimal threshold Dec 29 23:52:43.010117701 (584h23m42.881200736s) [seg_pos={0 0}, upload=512289]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:52:43.010192613 (584h23m42.881275648s) [upload=512289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:52:43.010198064 (584h23m42.881281099s) [upload=512289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:52:43.302225465 (584h23m43.1733085s) [upload=512289]: Done uploading segments... Dec 29 23:52:43.302243670 (584h23m43.173326705s) [upload=512289]: Upload finished. err: <nil> Duration: 8.036058011s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.547560447s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-29 23:54:42 UTC) orbiter warning: context deadline exceeded Duration: 3.547652132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..X..X....X.....X......XX...X...X..XXX............XXX....X.] duration: 6.055480342s trace: https://jaeger.dev.storj.io/trace/3ed94279973ea3fd FAIL (2025-12-29 23:53:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 23:52:40.396609305 (584h23m40.26769234s) [upload=512289]: Starting upload Dec 29 23:52:40.396695880 (584h23m40.267778915s) [upload=512289]: Done starting upload Dec 29 23:52:40.396802342 (584h23m40.267885377s) [upload=512289]: Uploading segments... Dec 29 23:52:40.397087595 (584h23m40.26817064s) [upload=512289]: Got next segment. Inline: false Dec 29 23:52:40.397160774 (584h23m40.26824381s) [upload=512289]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:52:40.397168290 (584h23m40.268251335s) [upload=512289]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:52:40.662507012 (584h23m40.533590047s) [seg_pos={0 0}, upload=512289]: Begin upload segment... Dec 29 23:52:40.662584950 (584h23m40.533667995s) [seg_pos={0 0}, upload=512289]: Done begin upload segment. Dec 29 23:52:40.662613565 (584h23m40.53369661s) [noise=true , address=145.79.227.34:4905 , piece_id=XAPWWVZFZOYUB4SIXZFVUNRW7MN3TELYNSVF7YBHR2LHGH7QIUBQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.663209670 (584h23m40.534292715s) [noise=true , address=45.91.116.14:4192 , piece_id=6R3COZT6RRLVBXZY6D6ZMYBVKDCFIJ2XC4LSHFUY4W437IRWSGYQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.663881980 (584h23m40.534965015s) [noise=true , address=145.79.227.14:4837 , piece_id=6OAWQS7BFFLL6GIFUJ6HYCYUK7XATDVGEIJD64EZIHMM4HMZFYHQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.664173766 (584h23m40.535256801s) [noise=true , address=145.79.227.33:4871 , piece_id=K4Q4N5JDEWF2BOM3MB2QYPJ37FB3VZOLDZESXPJGK4SVVTIVLU6A, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.664464128 (584h23m40.535547174s) [noise=true , address=145.79.227.39:4013 , piece_id=64DIJVBDVC577RN63PMQ2CJVYAIMWBFA4G7MKQXPAFQUL2LTPVHQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.664794538 (584h23m40.535877573s) [noise=true , address=145.79.227.12:4765 , piece_id=UGDO5GNOL7ZNOYOJUTBUYWFAJ5VA34TOTRBHXGETSOILOKF6BQGA, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.665126139 (584h23m40.536209184s) [noise=true , address=145.79.227.3:4150 , piece_id=EGR7H3B3P3P5UJD22HN2SFVC5QRMSJPH32QVMUK4RZPUCTTTQ7CA, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.666091417 (584h23m40.537174452s) [noise=true , address=145.79.227.18:4006 , piece_id=FFKXKMKFC2LWZSF24Z23KYPIXXJNUEWZ5BDOCH5BC6RCQWHBUZDQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.666243066 (584h23m40.537326101s) [noise=true , address=45.91.116.11:4117 , piece_id=MUMUQKWEIY5XD7HNPP5CQWU226V3CMCYV4QWBI3FJO3C5HW7TE2Q, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.667229253 (584h23m40.538312298s) [noise=true , address=45.91.116.10:4082 , piece_id=PLERXO5CDHLCODU6A4NT2I347PGFFOLWLW6OJCRZTOLZDVUDZSKA, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.667347840 (584h23m40.538430875s) [noise=true , address=145.79.227.16:4910 , piece_id=DB5I6XG6DCMDELUVBO4J4MDPMFWOSVZVBKHLTBIRKHMXKFN5UW6A, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.668268052 (584h23m40.539351097s) [noise=true , address=145.79.227.46:4602 , piece_id=JJGLE5CLZBNRVLHVATHFMP54AM3XQALOCYUGBWFRCDBEUXOIKGMQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.669317530 (584h23m40.540400565s) [noise=true , address=77.90.31.14:4663 , piece_id=FQPF4ELL2S5GUSGKJPACQ437KHO25ULWBQDVYWTU6D6KYKTI2R7Q, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.669704568 (584h23m40.540787603s) [noise=true , address=45.91.116.8:4047 , piece_id=UIYYVKGDUJOP5I54WB6C4ILHKWTPDPXF33UCWZO2VMTHUJVZZ7RA, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.671770611 (584h23m40.542853646s) [noise=true , address=145.79.227.17:4972 , piece_id=YKOXP4ICO3YS36XJBP3XDJHAXLFR7TPWAIQUWB7VLZPUIMLMBPWQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.672991927 (584h23m40.544074962s) [noise=true , address=145.79.227.38:4978 , piece_id=QCOLYC3U5BNCMQYZHQYFO2P37TXE5WHTVD6JC47N2NY65YI6FDBQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:42.272017614 (584h23m42.143100659s) [upload=512289]: Next returned nil segment Dec 29 23:52:42.272029707 (584h23m42.143112742s) [upload=512289]: Waiting for error group managing segments... Dec 29 23:52:42.653571787 (584h23m42.524654822s) [noise=true , address=145.79.227.18:4006 , piece_id=FFKXKMKFC2LWZSF24Z23KYPIXXJNUEWZ5BDOCH5BC6RCQWHBUZDQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.654325893 (584h23m42.525408928s) [noise=true , address=145.79.227.14:4837 , piece_id=6OAWQS7BFFLL6GIFUJ6HYCYUK7XATDVGEIJD64EZIHMM4HMZFYHQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.656840582 (584h23m42.527923627s) [noise=true , address=145.79.227.12:4765 , piece_id=UGDO5GNOL7ZNOYOJUTBUYWFAJ5VA34TOTRBHXGETSOILOKF6BQGA, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.673972653 (584h23m42.545055698s) [noise=true , address=145.79.227.3:4150 , piece_id=EGR7H3B3P3P5UJD22HN2SFVC5QRMSJPH32QVMUK4RZPUCTTTQ7CA, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.679012039 (584h23m42.550095074s) [noise=true , address=145.79.227.39:4013 , piece_id=64DIJVBDVC577RN63PMQ2CJVYAIMWBFA4G7MKQXPAFQUL2LTPVHQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.679304616 (584h23m42.550387651s) [noise=true , address=145.79.227.16:4910 , piece_id=DB5I6XG6DCMDELUVBO4J4MDPMFWOSVZVBKHLTBIRKHMXKFN5UW6A, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.680776659 (584h23m42.551859704s) [noise=true , address=145.79.227.38:4978 , piece_id=QCOLYC3U5BNCMQYZHQYFO2P37TXE5WHTVD6JC47N2NY65YI6FDBQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.682216040 (584h23m42.553299075s) [noise=true , address=145.79.227.17:4972 , piece_id=YKOXP4ICO3YS36XJBP3XDJHAXLFR7TPWAIQUWB7VLZPUIMLMBPWQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.843092541 (584h23m42.714175586s) [noise=true , address=77.90.31.14:4663 , piece_id=FQPF4ELL2S5GUSGKJPACQ437KHO25ULWBQDVYWTU6D6KYKTI2R7Q, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.913519134 (584h23m42.784602179s) [noise=true , address=45.91.116.10:4082 , piece_id=PLERXO5CDHLCODU6A4NT2I347PGFFOLWLW6OJCRZTOLZDVUDZSKA, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.916411311 (584h23m42.787494376s) [noise=true , address=45.91.116.11:4117 , piece_id=MUMUQKWEIY5XD7HNPP5CQWU226V3CMCYV4QWBI3FJO3C5HW7TE2Q, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.963330993 (584h23m42.834414028s) [noise=true , address=45.91.116.8:4047 , piece_id=UIYYVKGDUJOP5I54WB6C4ILHKWTPDPXF33UCWZO2VMTHUJVZZ7RA, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:43.009589565 (584h23m42.880672611s) [noise=true , address=45.91.116.14:4192 , piece_id=6R3COZT6RRLVBXZY6D6ZMYBVKDCFIJ2XC4LSHFUY4W437IRWSGYQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:43.009610375 (584h23m42.880693421s) [seg_pos={0 0}, upload=512289]: Segment reached optimal threshold of 13 pieces. Dec 29 23:52:43.009841385 (584h23m42.88092442s) [noise=true , address=145.79.227.33:4871 , piece_id=K4Q4N5JDEWF2BOM3MB2QYPJ37FB3VZOLDZESXPJGK4SVVTIVLU6A, seg_pos={0 0}, upload=512289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 29 23:52:43.009848288 (584h23m42.880931333s) [noise=true , address=145.79.227.33:4871 , piece_id=K4Q4N5JDEWF2BOM3MB2QYPJ37FB3VZOLDZESXPJGK4SVVTIVLU6A, seg_pos={0 0}, upload=512289]: Long tail cancellation due to optimal threshold Dec 29 23:52:43.009995278 (584h23m42.881078323s) [noise=true , address=145.79.227.34:4905 , piece_id=XAPWWVZFZOYUB4SIXZFVUNRW7MN3TELYNSVF7YBHR2LHGH7QIUBQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 29 23:52:43.010010898 (584h23m42.881093943s) [noise=true , address=145.79.227.34:4905 , piece_id=XAPWWVZFZOYUB4SIXZFVUNRW7MN3TELYNSVF7YBHR2LHGH7QIUBQ, seg_pos={0 0}, upload=512289]: Long tail cancellation due to optimal threshold Dec 29 23:52:43.010089697 (584h23m42.881172742s) [noise=true , address=145.79.227.46:4602 , piece_id=JJGLE5CLZBNRVLHVATHFMP54AM3XQALOCYUGBWFRCDBEUXOIKGMQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d): protocol: expected piece hash; context canceled; EOF Dec 29 23:52:43.010097262 (584h23m42.881180297s) [noise=true , address=145.79.227.46:4602 , piece_id=JJGLE5CLZBNRVLHVATHFMP54AM3XQALOCYUGBWFRCDBEUXOIKGMQ, seg_pos={0 0}, upload=512289]: Long tail cancellation due to optimal threshold Dec 29 23:52:43.010117701 (584h23m42.881200736s) [seg_pos={0 0}, upload=512289]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:52:43.010192613 (584h23m42.881275648s) [upload=512289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:52:43.010198064 (584h23m42.881281099s) [upload=512289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:52:43.302225465 (584h23m43.1733085s) [upload=512289]: Done uploading segments... Dec 29 23:52:43.302243670 (584h23m43.173326705s) [upload=512289]: Upload finished. err: <nil> Duration: 6.05550574s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.383771368s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-29 23:53:40 UTC) orbiter warning: context deadline exceeded Duration: 3.383798439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X..X..X....X.....X......XX...X...X..XXX............XXX....X] duration: 30.00264023s trace: https://jaeger.dev.storj.io/trace/3ed94279973ea3fd FAIL (2025-12-29 23:53:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 23:52:40.396609305 (584h23m40.26769234s) [upload=512289]: Starting upload Dec 29 23:52:40.396695880 (584h23m40.267778915s) [upload=512289]: Done starting upload Dec 29 23:52:40.396802342 (584h23m40.267885377s) [upload=512289]: Uploading segments... Dec 29 23:52:40.397087595 (584h23m40.26817064s) [upload=512289]: Got next segment. Inline: false Dec 29 23:52:40.397160774 (584h23m40.26824381s) [upload=512289]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:52:40.397168290 (584h23m40.268251335s) [upload=512289]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:52:40.662507012 (584h23m40.533590047s) [seg_pos={0 0}, upload=512289]: Begin upload segment... Dec 29 23:52:40.662584950 (584h23m40.533667995s) [seg_pos={0 0}, upload=512289]: Done begin upload segment. Dec 29 23:52:40.662613565 (584h23m40.53369661s) [noise=true , address=145.79.227.34:4905 , piece_id=XAPWWVZFZOYUB4SIXZFVUNRW7MN3TELYNSVF7YBHR2LHGH7QIUBQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.663209670 (584h23m40.534292715s) [noise=true , address=45.91.116.14:4192 , piece_id=6R3COZT6RRLVBXZY6D6ZMYBVKDCFIJ2XC4LSHFUY4W437IRWSGYQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.663881980 (584h23m40.534965015s) [noise=true , address=145.79.227.14:4837 , piece_id=6OAWQS7BFFLL6GIFUJ6HYCYUK7XATDVGEIJD64EZIHMM4HMZFYHQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.664173766 (584h23m40.535256801s) [noise=true , address=145.79.227.33:4871 , piece_id=K4Q4N5JDEWF2BOM3MB2QYPJ37FB3VZOLDZESXPJGK4SVVTIVLU6A, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.664464128 (584h23m40.535547174s) [noise=true , address=145.79.227.39:4013 , piece_id=64DIJVBDVC577RN63PMQ2CJVYAIMWBFA4G7MKQXPAFQUL2LTPVHQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.664794538 (584h23m40.535877573s) [noise=true , address=145.79.227.12:4765 , piece_id=UGDO5GNOL7ZNOYOJUTBUYWFAJ5VA34TOTRBHXGETSOILOKF6BQGA, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.665126139 (584h23m40.536209184s) [noise=true , address=145.79.227.3:4150 , piece_id=EGR7H3B3P3P5UJD22HN2SFVC5QRMSJPH32QVMUK4RZPUCTTTQ7CA, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.666091417 (584h23m40.537174452s) [noise=true , address=145.79.227.18:4006 , piece_id=FFKXKMKFC2LWZSF24Z23KYPIXXJNUEWZ5BDOCH5BC6RCQWHBUZDQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.666243066 (584h23m40.537326101s) [noise=true , address=45.91.116.11:4117 , piece_id=MUMUQKWEIY5XD7HNPP5CQWU226V3CMCYV4QWBI3FJO3C5HW7TE2Q, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.667229253 (584h23m40.538312298s) [noise=true , address=45.91.116.10:4082 , piece_id=PLERXO5CDHLCODU6A4NT2I347PGFFOLWLW6OJCRZTOLZDVUDZSKA, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.667347840 (584h23m40.538430875s) [noise=true , address=145.79.227.16:4910 , piece_id=DB5I6XG6DCMDELUVBO4J4MDPMFWOSVZVBKHLTBIRKHMXKFN5UW6A, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.668268052 (584h23m40.539351097s) [noise=true , address=145.79.227.46:4602 , piece_id=JJGLE5CLZBNRVLHVATHFMP54AM3XQALOCYUGBWFRCDBEUXOIKGMQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.669317530 (584h23m40.540400565s) [noise=true , address=77.90.31.14:4663 , piece_id=FQPF4ELL2S5GUSGKJPACQ437KHO25ULWBQDVYWTU6D6KYKTI2R7Q, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.669704568 (584h23m40.540787603s) [noise=true , address=45.91.116.8:4047 , piece_id=UIYYVKGDUJOP5I54WB6C4ILHKWTPDPXF33UCWZO2VMTHUJVZZ7RA, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.671770611 (584h23m40.542853646s) [noise=true , address=145.79.227.17:4972 , piece_id=YKOXP4ICO3YS36XJBP3XDJHAXLFR7TPWAIQUWB7VLZPUIMLMBPWQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:40.672991927 (584h23m40.544074962s) [noise=true , address=145.79.227.38:4978 , piece_id=QCOLYC3U5BNCMQYZHQYFO2P37TXE5WHTVD6JC47N2NY65YI6FDBQ, seg_pos={0 0}, upload=512289]: Uploading piece... Dec 29 23:52:42.272017614 (584h23m42.143100659s) [upload=512289]: Next returned nil segment Dec 29 23:52:42.272029707 (584h23m42.143112742s) [upload=512289]: Waiting for error group managing segments... Dec 29 23:52:42.653571787 (584h23m42.524654822s) [noise=true , address=145.79.227.18:4006 , piece_id=FFKXKMKFC2LWZSF24Z23KYPIXXJNUEWZ5BDOCH5BC6RCQWHBUZDQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.654325893 (584h23m42.525408928s) [noise=true , address=145.79.227.14:4837 , piece_id=6OAWQS7BFFLL6GIFUJ6HYCYUK7XATDVGEIJD64EZIHMM4HMZFYHQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.656840582 (584h23m42.527923627s) [noise=true , address=145.79.227.12:4765 , piece_id=UGDO5GNOL7ZNOYOJUTBUYWFAJ5VA34TOTRBHXGETSOILOKF6BQGA, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.673972653 (584h23m42.545055698s) [noise=true , address=145.79.227.3:4150 , piece_id=EGR7H3B3P3P5UJD22HN2SFVC5QRMSJPH32QVMUK4RZPUCTTTQ7CA, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.679012039 (584h23m42.550095074s) [noise=true , address=145.79.227.39:4013 , piece_id=64DIJVBDVC577RN63PMQ2CJVYAIMWBFA4G7MKQXPAFQUL2LTPVHQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.679304616 (584h23m42.550387651s) [noise=true , address=145.79.227.16:4910 , piece_id=DB5I6XG6DCMDELUVBO4J4MDPMFWOSVZVBKHLTBIRKHMXKFN5UW6A, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.680776659 (584h23m42.551859704s) [noise=true , address=145.79.227.38:4978 , piece_id=QCOLYC3U5BNCMQYZHQYFO2P37TXE5WHTVD6JC47N2NY65YI6FDBQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.682216040 (584h23m42.553299075s) [noise=true , address=145.79.227.17:4972 , piece_id=YKOXP4ICO3YS36XJBP3XDJHAXLFR7TPWAIQUWB7VLZPUIMLMBPWQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.843092541 (584h23m42.714175586s) [noise=true , address=77.90.31.14:4663 , piece_id=FQPF4ELL2S5GUSGKJPACQ437KHO25ULWBQDVYWTU6D6KYKTI2R7Q, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.913519134 (584h23m42.784602179s) [noise=true , address=45.91.116.10:4082 , piece_id=PLERXO5CDHLCODU6A4NT2I347PGFFOLWLW6OJCRZTOLZDVUDZSKA, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.916411311 (584h23m42.787494376s) [noise=true , address=45.91.116.11:4117 , piece_id=MUMUQKWEIY5XD7HNPP5CQWU226V3CMCYV4QWBI3FJO3C5HW7TE2Q, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:42.963330993 (584h23m42.834414028s) [noise=true , address=45.91.116.8:4047 , piece_id=UIYYVKGDUJOP5I54WB6C4ILHKWTPDPXF33UCWZO2VMTHUJVZZ7RA, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:43.009589565 (584h23m42.880672611s) [noise=true , address=45.91.116.14:4192 , piece_id=6R3COZT6RRLVBXZY6D6ZMYBVKDCFIJ2XC4LSHFUY4W437IRWSGYQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: <nil> Dec 29 23:52:43.009610375 (584h23m42.880693421s) [seg_pos={0 0}, upload=512289]: Segment reached optimal threshold of 13 pieces. Dec 29 23:52:43.009841385 (584h23m42.88092442s) [noise=true , address=145.79.227.33:4871 , piece_id=K4Q4N5JDEWF2BOM3MB2QYPJ37FB3VZOLDZESXPJGK4SVVTIVLU6A, seg_pos={0 0}, upload=512289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 29 23:52:43.009848288 (584h23m42.880931333s) [noise=true , address=145.79.227.33:4871 , piece_id=K4Q4N5JDEWF2BOM3MB2QYPJ37FB3VZOLDZESXPJGK4SVVTIVLU6A, seg_pos={0 0}, upload=512289]: Long tail cancellation due to optimal threshold Dec 29 23:52:43.009995278 (584h23m42.881078323s) [noise=true , address=145.79.227.34:4905 , piece_id=XAPWWVZFZOYUB4SIXZFVUNRW7MN3TELYNSVF7YBHR2LHGH7QIUBQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 29 23:52:43.010010898 (584h23m42.881093943s) [noise=true , address=145.79.227.34:4905 , piece_id=XAPWWVZFZOYUB4SIXZFVUNRW7MN3TELYNSVF7YBHR2LHGH7QIUBQ, seg_pos={0 0}, upload=512289]: Long tail cancellation due to optimal threshold Dec 29 23:52:43.010089697 (584h23m42.881172742s) [noise=true , address=145.79.227.46:4602 , piece_id=JJGLE5CLZBNRVLHVATHFMP54AM3XQALOCYUGBWFRCDBEUXOIKGMQ, seg_pos={0 0}, upload=512289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d): protocol: expected piece hash; context canceled; EOF Dec 29 23:52:43.010097262 (584h23m42.881180297s) [noise=true , address=145.79.227.46:4602 , piece_id=JJGLE5CLZBNRVLHVATHFMP54AM3XQALOCYUGBWFRCDBEUXOIKGMQ, seg_pos={0 0}, upload=512289]: Long tail cancellation due to optimal threshold Dec 29 23:52:43.010117701 (584h23m42.881200736s) [seg_pos={0 0}, upload=512289]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:52:43.010192613 (584h23m42.881275648s) [upload=512289]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:52:43.010198064 (584h23m42.881281099s) [upload=512289]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:52:43.302225465 (584h23m43.1733085s) [upload=512289]: Done uploading segments... Dec 29 23:52:43.302243670 (584h23m43.173326705s) [upload=512289]: Upload finished. err: <nil> Duration: 30.002663143s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.415362933s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-29 23:52:31 UTC) orbiter warning: context deadline exceeded Duration: 3.415389111s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X..X..X....X.....X......XX...X...X..XXX............XXX....] duration: 5.900832691s trace: https://jaeger.dev.storj.io/trace/18fd4c88bd813274 FAIL (2025-12-29 23:51:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :48:31.441307553 (584h19m31.312390588s) [upload=512232]: Starting upload Dec 29 23:48:31.441363339 (584h19m31.312446374s) [upload=512232]: Done starting upload Dec 29 23:48:31.441492194 (584h19m31.312575249s) [upload=512232]: Uploading segments... Dec 29 23:48:31.441658791 (584h19m31.312741836s) [upload=512232]: Got next segment. Inline: false Dec 29 23:48:31.441708046 (584h19m31.312791081s) [upload=512232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:48:31.441713927 (584h19m31.312796972s) [upload=512232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:48:31.691578184 (584h19m31.562661239s) [seg_pos={0 0}, upload=512232]: Begin upload segment... Dec 29 23:48:31.691637777 (584h19m31.562720812s) [seg_pos={0 0}, upload=512232]: Done begin upload segment. Dec 29 23:48:31.691688784 (584h19m31.562771819s) [noise=true , address=145.79.227.6:4258 , piece_id=OYJTQUX5FQQ6DTWKBU6BFQ3X5USCH3GBOUVDAXHRCZBHTILN2ZDQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.691700456 (584h19m31.562783501s) [noise=true , address=145.79.227.47:4267 , piece_id=FBNNU4PX5P4DFQU5YC53GE353URBAYAXS5RQGOOUI6KOVRKNMSHA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.691814223 (584h19m31.562897268s) [noise=true , address=77.90.31.14:4661 , piece_id=CIGRDFSNKNEMGJI5GTLLJ22Y3TR3QWSTQWLAVKPPHOYFOUZO6F6Q, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.692637470 (584h19m31.563720515s) [noise=true , address=145.79.227.45:4231 , piece_id=MICXNPBWQ6ZHWPL2CDB4KOH7GC3LSZJW2YXKVQJXCMENM6FGLZCQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.692653541 (584h19m31.563736586s) [noise=true , address=45.91.116.12:4225 , piece_id=2X24WJOLZZMWGH7RHZHW76XOJXQVTCG47DGSBNDN7TV5TP4GE4CA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.692704107 (584h19m31.563787162s) [noise=true , address=145.79.227.13:4804 , piece_id=QLGTAHKHVNKJDB3RAKRSJXYFLAQ5ZAW4JEDQENZ26KPPM62WUSXA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.693338315 (584h19m31.56442135s) [noise=true , address=145.79.227.11:4732 , piece_id=AXLV6RGTC3OWYE3AUZSOPYF6R62GTA27KTCOYWEOPHYXMJQPW6KQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.693758996 (584h19m31.564842031s) [noise=true , address=145.79.227.26:4653 , piece_id=II3A7L3PBV45OEZIPXTVW42LOJXXPZUKF6NHBPJ2WZLZLYCEHFZQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.694283244 (584h19m31.565366289s) [noise=true , address=45.91.116.6:4974 , piece_id=Z6YVJSGV5MTKIM5C5USKETVRY6UQOIRWSPBPHTHN37X64PAWHZAA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.694694106 (584h19m31.565777141s) [noise=true , address=145.79.227.30:4762 , piece_id=GR5MLPLWVBYQABLPQPUTMMUHIQQIZ32UOUTRMKF6JNG4NTPV52GA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.694821569 (584h19m31.565904604s) [noise=true , address=145.79.227.17:4972 , piece_id=S27WMVQVEQWYOFPVMWCPWEZN5YX7MI6JQSX56W3Z3RI3TX5T7G5A, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.695283439 (584h19m31.566366474s) [noise=true , address=145.79.227.5:4223 , piece_id=4QNWFLOW344XAN4QAFUQ7QOY2N764CB5HDY4Z52VL4HDMMNCKTIA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.695713587 (584h19m31.566796622s) [noise=true , address=77.90.31.10:4518 , piece_id=RFFICLSMGOV5XHHHKDCO4TG7RAH3D4AYZCPBP65JPCIAD7VL2EUA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.695998771 (584h19m31.567081816s) [noise=true , address=145.79.227.7:4294 , piece_id=VVKZ5GNKZAMHXRD6GTU5EUW27QNDHSRNC6W4Y6C6XX5SGS6GTONQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.696453337 (584h19m31.567536382s) [noise=true , address=145.79.227.3:4149 , piece_id=UREXLSG3A4XDPSH6TMFTVFTYBMI6EAMK2OXFLLG6TONP7OWXC72Q, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.697923306 (584h19m31.569006341s) [noise=true , address=145.79.227.14:4839 , piece_id=64JYSB7F4SOWFAJD4LPBLGI3DI6W3SNNSVB4JDDINU3HUAEC73HA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:33.390825229 (584h19m33.261908264s) [upload=512232]: Next returned nil segment Dec 29 23:48:33.390837111 (584h19m33.261920146s) [upload=512232]: Waiting for error group managing segments... Dec 29 23:48:33.663774052 (584h19m33.534857087s) [noise=true , address=145.79.227.3:4149 , piece_id=UREXLSG3A4XDPSH6TMFTVFTYBMI6EAMK2OXFLLG6TONP7OWXC72Q, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.668648524 (584h19m33.539731559s) [noise=true , address=145.79.227.11:4732 , piece_id=AXLV6RGTC3OWYE3AUZSOPYF6R62GTA27KTCOYWEOPHYXMJQPW6KQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.673561750 (584h19m33.544644785s) [noise=true , address=145.79.227.26:4653 , piece_id=II3A7L3PBV45OEZIPXTVW42LOJXXPZUKF6NHBPJ2WZLZLYCEHFZQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.673865738 (584h19m33.544948813s) [noise=true , address=145.79.227.14:4839 , piece_id=64JYSB7F4SOWFAJD4LPBLGI3DI6W3SNNSVB4JDDINU3HUAEC73HA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.677295810 (584h19m33.548378845s) [noise=true , address=145.79.227.6:4258 , piece_id=OYJTQUX5FQQ6DTWKBU6BFQ3X5USCH3GBOUVDAXHRCZBHTILN2ZDQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.684755484 (584h19m33.555838519s) [noise=true , address=145.79.227.45:4231 , piece_id=MICXNPBWQ6ZHWPL2CDB4KOH7GC3LSZJW2YXKVQJXCMENM6FGLZCQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.696615256 (584h19m33.567698291s) [noise=true , address=145.79.227.13:4804 , piece_id=QLGTAHKHVNKJDB3RAKRSJXYFLAQ5ZAW4JEDQENZ26KPPM62WUSXA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.700671430 (584h19m33.571754465s) [noise=true , address=145.79.227.7:4294 , piece_id=VVKZ5GNKZAMHXRD6GTU5EUW27QNDHSRNC6W4Y6C6XX5SGS6GTONQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.716500491 (584h19m33.587583536s) [noise=true , address=145.79.227.17:4972 , piece_id=S27WMVQVEQWYOFPVMWCPWEZN5YX7MI6JQSX56W3Z3RI3TX5T7G5A, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.723906112 (584h19m33.594989157s) [noise=true , address=145.79.227.47:4267 , piece_id=FBNNU4PX5P4DFQU5YC53GE353URBAYAXS5RQGOOUI6KOVRKNMSHA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.863810019 (584h19m33.734893054s) [noise=true , address=77.90.31.10:4518 , piece_id=RFFICLSMGOV5XHHHKDCO4TG7RAH3D4AYZCPBP65JPCIAD7VL2EUA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.887264209 (584h19m33.758347244s) [noise=true , address=77.90.31.14:4661 , piece_id=CIGRDFSNKNEMGJI5GTLLJ22Y3TR3QWSTQWLAVKPPHOYFOUZO6F6Q, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.970123234 (584h19m33.841206269s) [noise=true , address=45.91.116.12:4225 , piece_id=2X24WJOLZZMWGH7RHZHW76XOJXQVTCG47DGSBNDN7TV5TP4GE4CA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.970145396 (584h19m33.841228441s) [seg_pos={0 0}, upload=512232]: Segment reached optimal threshold of 13 pieces. Dec 29 23:48:33.970410942 (584h19m33.841493977s) [noise=true , address=145.79.227.5:4223 , piece_id=4QNWFLOW344XAN4QAFUQ7QOY2N764CB5HDY4Z52VL4HDMMNCKTIA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:52288->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:52288->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52288->145.79.227.5:4223: use of closed network connection Dec 29 23:48:33.970420540 (584h19m33.841503575s) [noise=true , address=145.79.227.5:4223 , piece_id=4QNWFLOW344XAN4QAFUQ7QOY2N764CB5HDY4Z52VL4HDMMNCKTIA, seg_pos={0 0}, upload=512232]: Long tail cancellation due to optimal threshold Dec 29 23:48:33.970494230 (584h19m33.841577265s) [noise=true , address=145.79.227.30:4762 , piece_id=GR5MLPLWVBYQABLPQPUTMMUHIQQIZ32UOUTRMKF6JNG4NTPV52GA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 23:48:33.970509861 (584h19m33.841592906s) [noise=true , address=145.79.227.30:4762 , piece_id=GR5MLPLWVBYQABLPQPUTMMUHIQQIZ32UOUTRMKF6JNG4NTPV52GA, seg_pos={0 0}, upload=512232]: Long tail cancellation due to optimal threshold Dec 29 23:48:33.970640348 (584h19m33.841723384s) [noise=true , address=45.91.116.6:4974 , piece_id=Z6YVJSGV5MTKIM5C5USKETVRY6UQOIRWSPBPHTHN37X64PAWHZAA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 23:48:33.970645799 (584h19m33.841728844s) [noise=true , address=45.91.116.6:4974 , piece_id=Z6YVJSGV5MTKIM5C5USKETVRY6UQOIRWSPBPHTHN37X64PAWHZAA, seg_pos={0 0}, upload=512232]: Long tail cancellation due to optimal threshold Dec 29 23:48:33.970667961 (584h19m33.841750996s) [seg_pos={0 0}, upload=512232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:48:33.970741080 (584h19m33.841824115s) [upload=512232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:48:33.970745709 (584h19m33.841828744s) [upload=512232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:48:34.278829755 (584h19m34.14991279s) [upload=512232]: Done uploading segments... Dec 29 23:48:34.278849203 (584h19m34.149932248s) [upload=512232]: Upload finished. err: <nil> Duration: 5.900930637s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.5241162s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-29 23:51:45 UTC) orbiter warning: context deadline exceeded Duration: 3.524145626s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.514883196s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-29 23:51:21 UTC) orbiter warning: context deadline exceeded Duration: 3.514912902s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.552114708s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-29 23:50:59 UTC) orbiter warning: context deadline exceeded Duration: 3.552179501s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X..X..X....X.....X......XX...X...X..XXX............XXX...] duration: 6.1295135s trace: https://jaeger.dev.storj.io/trace/18fd4c88bd813274 FAIL (2025-12-29 23:50:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :48:31.441307553 (584h19m31.312390588s) [upload=512232]: Starting upload Dec 29 23:48:31.441363339 (584h19m31.312446374s) [upload=512232]: Done starting upload Dec 29 23:48:31.441492194 (584h19m31.312575249s) [upload=512232]: Uploading segments... Dec 29 23:48:31.441658791 (584h19m31.312741836s) [upload=512232]: Got next segment. Inline: false Dec 29 23:48:31.441708046 (584h19m31.312791081s) [upload=512232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:48:31.441713927 (584h19m31.312796972s) [upload=512232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:48:31.691578184 (584h19m31.562661239s) [seg_pos={0 0}, upload=512232]: Begin upload segment... Dec 29 23:48:31.691637777 (584h19m31.562720812s) [seg_pos={0 0}, upload=512232]: Done begin upload segment. Dec 29 23:48:31.691688784 (584h19m31.562771819s) [noise=true , address=145.79.227.6:4258 , piece_id=OYJTQUX5FQQ6DTWKBU6BFQ3X5USCH3GBOUVDAXHRCZBHTILN2ZDQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.691700456 (584h19m31.562783501s) [noise=true , address=145.79.227.47:4267 , piece_id=FBNNU4PX5P4DFQU5YC53GE353URBAYAXS5RQGOOUI6KOVRKNMSHA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.691814223 (584h19m31.562897268s) [noise=true , address=77.90.31.14:4661 , piece_id=CIGRDFSNKNEMGJI5GTLLJ22Y3TR3QWSTQWLAVKPPHOYFOUZO6F6Q, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.692637470 (584h19m31.563720515s) [noise=true , address=145.79.227.45:4231 , piece_id=MICXNPBWQ6ZHWPL2CDB4KOH7GC3LSZJW2YXKVQJXCMENM6FGLZCQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.692653541 (584h19m31.563736586s) [noise=true , address=45.91.116.12:4225 , piece_id=2X24WJOLZZMWGH7RHZHW76XOJXQVTCG47DGSBNDN7TV5TP4GE4CA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.692704107 (584h19m31.563787162s) [noise=true , address=145.79.227.13:4804 , piece_id=QLGTAHKHVNKJDB3RAKRSJXYFLAQ5ZAW4JEDQENZ26KPPM62WUSXA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.693338315 (584h19m31.56442135s) [noise=true , address=145.79.227.11:4732 , piece_id=AXLV6RGTC3OWYE3AUZSOPYF6R62GTA27KTCOYWEOPHYXMJQPW6KQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.693758996 (584h19m31.564842031s) [noise=true , address=145.79.227.26:4653 , piece_id=II3A7L3PBV45OEZIPXTVW42LOJXXPZUKF6NHBPJ2WZLZLYCEHFZQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.694283244 (584h19m31.565366289s) [noise=true , address=45.91.116.6:4974 , piece_id=Z6YVJSGV5MTKIM5C5USKETVRY6UQOIRWSPBPHTHN37X64PAWHZAA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.694694106 (584h19m31.565777141s) [noise=true , address=145.79.227.30:4762 , piece_id=GR5MLPLWVBYQABLPQPUTMMUHIQQIZ32UOUTRMKF6JNG4NTPV52GA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.694821569 (584h19m31.565904604s) [noise=true , address=145.79.227.17:4972 , piece_id=S27WMVQVEQWYOFPVMWCPWEZN5YX7MI6JQSX56W3Z3RI3TX5T7G5A, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.695283439 (584h19m31.566366474s) [noise=true , address=145.79.227.5:4223 , piece_id=4QNWFLOW344XAN4QAFUQ7QOY2N764CB5HDY4Z52VL4HDMMNCKTIA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.695713587 (584h19m31.566796622s) [noise=true , address=77.90.31.10:4518 , piece_id=RFFICLSMGOV5XHHHKDCO4TG7RAH3D4AYZCPBP65JPCIAD7VL2EUA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.695998771 (584h19m31.567081816s) [noise=true , address=145.79.227.7:4294 , piece_id=VVKZ5GNKZAMHXRD6GTU5EUW27QNDHSRNC6W4Y6C6XX5SGS6GTONQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.696453337 (584h19m31.567536382s) [noise=true , address=145.79.227.3:4149 , piece_id=UREXLSG3A4XDPSH6TMFTVFTYBMI6EAMK2OXFLLG6TONP7OWXC72Q, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.697923306 (584h19m31.569006341s) [noise=true , address=145.79.227.14:4839 , piece_id=64JYSB7F4SOWFAJD4LPBLGI3DI6W3SNNSVB4JDDINU3HUAEC73HA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:33.390825229 (584h19m33.261908264s) [upload=512232]: Next returned nil segment Dec 29 23:48:33.390837111 (584h19m33.261920146s) [upload=512232]: Waiting for error group managing segments... Dec 29 23:48:33.663774052 (584h19m33.534857087s) [noise=true , address=145.79.227.3:4149 , piece_id=UREXLSG3A4XDPSH6TMFTVFTYBMI6EAMK2OXFLLG6TONP7OWXC72Q, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.668648524 (584h19m33.539731559s) [noise=true , address=145.79.227.11:4732 , piece_id=AXLV6RGTC3OWYE3AUZSOPYF6R62GTA27KTCOYWEOPHYXMJQPW6KQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.673561750 (584h19m33.544644785s) [noise=true , address=145.79.227.26:4653 , piece_id=II3A7L3PBV45OEZIPXTVW42LOJXXPZUKF6NHBPJ2WZLZLYCEHFZQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.673865738 (584h19m33.544948813s) [noise=true , address=145.79.227.14:4839 , piece_id=64JYSB7F4SOWFAJD4LPBLGI3DI6W3SNNSVB4JDDINU3HUAEC73HA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.677295810 (584h19m33.548378845s) [noise=true , address=145.79.227.6:4258 , piece_id=OYJTQUX5FQQ6DTWKBU6BFQ3X5USCH3GBOUVDAXHRCZBHTILN2ZDQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.684755484 (584h19m33.555838519s) [noise=true , address=145.79.227.45:4231 , piece_id=MICXNPBWQ6ZHWPL2CDB4KOH7GC3LSZJW2YXKVQJXCMENM6FGLZCQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.696615256 (584h19m33.567698291s) [noise=true , address=145.79.227.13:4804 , piece_id=QLGTAHKHVNKJDB3RAKRSJXYFLAQ5ZAW4JEDQENZ26KPPM62WUSXA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.700671430 (584h19m33.571754465s) [noise=true , address=145.79.227.7:4294 , piece_id=VVKZ5GNKZAMHXRD6GTU5EUW27QNDHSRNC6W4Y6C6XX5SGS6GTONQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.716500491 (584h19m33.587583536s) [noise=true , address=145.79.227.17:4972 , piece_id=S27WMVQVEQWYOFPVMWCPWEZN5YX7MI6JQSX56W3Z3RI3TX5T7G5A, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.723906112 (584h19m33.594989157s) [noise=true , address=145.79.227.47:4267 , piece_id=FBNNU4PX5P4DFQU5YC53GE353URBAYAXS5RQGOOUI6KOVRKNMSHA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.863810019 (584h19m33.734893054s) [noise=true , address=77.90.31.10:4518 , piece_id=RFFICLSMGOV5XHHHKDCO4TG7RAH3D4AYZCPBP65JPCIAD7VL2EUA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.887264209 (584h19m33.758347244s) [noise=true , address=77.90.31.14:4661 , piece_id=CIGRDFSNKNEMGJI5GTLLJ22Y3TR3QWSTQWLAVKPPHOYFOUZO6F6Q, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.970123234 (584h19m33.841206269s) [noise=true , address=45.91.116.12:4225 , piece_id=2X24WJOLZZMWGH7RHZHW76XOJXQVTCG47DGSBNDN7TV5TP4GE4CA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.970145396 (584h19m33.841228441s) [seg_pos={0 0}, upload=512232]: Segment reached optimal threshold of 13 pieces. Dec 29 23:48:33.970410942 (584h19m33.841493977s) [noise=true , address=145.79.227.5:4223 , piece_id=4QNWFLOW344XAN4QAFUQ7QOY2N764CB5HDY4Z52VL4HDMMNCKTIA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:52288->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:52288->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52288->145.79.227.5:4223: use of closed network connection Dec 29 23:48:33.970420540 (584h19m33.841503575s) [noise=true , address=145.79.227.5:4223 , piece_id=4QNWFLOW344XAN4QAFUQ7QOY2N764CB5HDY4Z52VL4HDMMNCKTIA, seg_pos={0 0}, upload=512232]: Long tail cancellation due to optimal threshold Dec 29 23:48:33.970494230 (584h19m33.841577265s) [noise=true , address=145.79.227.30:4762 , piece_id=GR5MLPLWVBYQABLPQPUTMMUHIQQIZ32UOUTRMKF6JNG4NTPV52GA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 23:48:33.970509861 (584h19m33.841592906s) [noise=true , address=145.79.227.30:4762 , piece_id=GR5MLPLWVBYQABLPQPUTMMUHIQQIZ32UOUTRMKF6JNG4NTPV52GA, seg_pos={0 0}, upload=512232]: Long tail cancellation due to optimal threshold Dec 29 23:48:33.970640348 (584h19m33.841723384s) [noise=true , address=45.91.116.6:4974 , piece_id=Z6YVJSGV5MTKIM5C5USKETVRY6UQOIRWSPBPHTHN37X64PAWHZAA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 23:48:33.970645799 (584h19m33.841728844s) [noise=true , address=45.91.116.6:4974 , piece_id=Z6YVJSGV5MTKIM5C5USKETVRY6UQOIRWSPBPHTHN37X64PAWHZAA, seg_pos={0 0}, upload=512232]: Long tail cancellation due to optimal threshold Dec 29 23:48:33.970667961 (584h19m33.841750996s) [seg_pos={0 0}, upload=512232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:48:33.970741080 (584h19m33.841824115s) [upload=512232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:48:33.970745709 (584h19m33.841828744s) [upload=512232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:48:34.278829755 (584h19m34.14991279s) [upload=512232]: Done uploading segments... Dec 29 23:48:34.278849203 (584h19m34.149932248s) [upload=512232]: Upload finished. err: <nil> Duration: 6.1295295s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X..X..X....X.....X......XX...X...X..XXX............XXX..] duration: 24.636607947s trace: https://jaeger.dev.storj.io/trace/18fd4c88bd813274 FAIL (2025-12-29 23:50:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :48:31.441307553 (584h19m31.312390588s) [upload=512232]: Starting upload Dec 29 23:48:31.441363339 (584h19m31.312446374s) [upload=512232]: Done starting upload Dec 29 23:48:31.441492194 (584h19m31.312575249s) [upload=512232]: Uploading segments... Dec 29 23:48:31.441658791 (584h19m31.312741836s) [upload=512232]: Got next segment. Inline: false Dec 29 23:48:31.441708046 (584h19m31.312791081s) [upload=512232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:48:31.441713927 (584h19m31.312796972s) [upload=512232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:48:31.691578184 (584h19m31.562661239s) [seg_pos={0 0}, upload=512232]: Begin upload segment... Dec 29 23:48:31.691637777 (584h19m31.562720812s) [seg_pos={0 0}, upload=512232]: Done begin upload segment. Dec 29 23:48:31.691688784 (584h19m31.562771819s) [noise=true , address=145.79.227.6:4258 , piece_id=OYJTQUX5FQQ6DTWKBU6BFQ3X5USCH3GBOUVDAXHRCZBHTILN2ZDQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.691700456 (584h19m31.562783501s) [noise=true , address=145.79.227.47:4267 , piece_id=FBNNU4PX5P4DFQU5YC53GE353URBAYAXS5RQGOOUI6KOVRKNMSHA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.691814223 (584h19m31.562897268s) [noise=true , address=77.90.31.14:4661 , piece_id=CIGRDFSNKNEMGJI5GTLLJ22Y3TR3QWSTQWLAVKPPHOYFOUZO6F6Q, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.692637470 (584h19m31.563720515s) [noise=true , address=145.79.227.45:4231 , piece_id=MICXNPBWQ6ZHWPL2CDB4KOH7GC3LSZJW2YXKVQJXCMENM6FGLZCQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.692653541 (584h19m31.563736586s) [noise=true , address=45.91.116.12:4225 , piece_id=2X24WJOLZZMWGH7RHZHW76XOJXQVTCG47DGSBNDN7TV5TP4GE4CA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.692704107 (584h19m31.563787162s) [noise=true , address=145.79.227.13:4804 , piece_id=QLGTAHKHVNKJDB3RAKRSJXYFLAQ5ZAW4JEDQENZ26KPPM62WUSXA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.693338315 (584h19m31.56442135s) [noise=true , address=145.79.227.11:4732 , piece_id=AXLV6RGTC3OWYE3AUZSOPYF6R62GTA27KTCOYWEOPHYXMJQPW6KQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.693758996 (584h19m31.564842031s) [noise=true , address=145.79.227.26:4653 , piece_id=II3A7L3PBV45OEZIPXTVW42LOJXXPZUKF6NHBPJ2WZLZLYCEHFZQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.694283244 (584h19m31.565366289s) [noise=true , address=45.91.116.6:4974 , piece_id=Z6YVJSGV5MTKIM5C5USKETVRY6UQOIRWSPBPHTHN37X64PAWHZAA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.694694106 (584h19m31.565777141s) [noise=true , address=145.79.227.30:4762 , piece_id=GR5MLPLWVBYQABLPQPUTMMUHIQQIZ32UOUTRMKF6JNG4NTPV52GA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.694821569 (584h19m31.565904604s) [noise=true , address=145.79.227.17:4972 , piece_id=S27WMVQVEQWYOFPVMWCPWEZN5YX7MI6JQSX56W3Z3RI3TX5T7G5A, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.695283439 (584h19m31.566366474s) [noise=true , address=145.79.227.5:4223 , piece_id=4QNWFLOW344XAN4QAFUQ7QOY2N764CB5HDY4Z52VL4HDMMNCKTIA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.695713587 (584h19m31.566796622s) [noise=true , address=77.90.31.10:4518 , piece_id=RFFICLSMGOV5XHHHKDCO4TG7RAH3D4AYZCPBP65JPCIAD7VL2EUA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.695998771 (584h19m31.567081816s) [noise=true , address=145.79.227.7:4294 , piece_id=VVKZ5GNKZAMHXRD6GTU5EUW27QNDHSRNC6W4Y6C6XX5SGS6GTONQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.696453337 (584h19m31.567536382s) [noise=true , address=145.79.227.3:4149 , piece_id=UREXLSG3A4XDPSH6TMFTVFTYBMI6EAMK2OXFLLG6TONP7OWXC72Q, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.697923306 (584h19m31.569006341s) [noise=true , address=145.79.227.14:4839 , piece_id=64JYSB7F4SOWFAJD4LPBLGI3DI6W3SNNSVB4JDDINU3HUAEC73HA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:33.390825229 (584h19m33.261908264s) [upload=512232]: Next returned nil segment Dec 29 23:48:33.390837111 (584h19m33.261920146s) [upload=512232]: Waiting for error group managing segments... Dec 29 23:48:33.663774052 (584h19m33.534857087s) [noise=true , address=145.79.227.3:4149 , piece_id=UREXLSG3A4XDPSH6TMFTVFTYBMI6EAMK2OXFLLG6TONP7OWXC72Q, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.668648524 (584h19m33.539731559s) [noise=true , address=145.79.227.11:4732 , piece_id=AXLV6RGTC3OWYE3AUZSOPYF6R62GTA27KTCOYWEOPHYXMJQPW6KQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.673561750 (584h19m33.544644785s) [noise=true , address=145.79.227.26:4653 , piece_id=II3A7L3PBV45OEZIPXTVW42LOJXXPZUKF6NHBPJ2WZLZLYCEHFZQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.673865738 (584h19m33.544948813s) [noise=true , address=145.79.227.14:4839 , piece_id=64JYSB7F4SOWFAJD4LPBLGI3DI6W3SNNSVB4JDDINU3HUAEC73HA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.677295810 (584h19m33.548378845s) [noise=true , address=145.79.227.6:4258 , piece_id=OYJTQUX5FQQ6DTWKBU6BFQ3X5USCH3GBOUVDAXHRCZBHTILN2ZDQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.684755484 (584h19m33.555838519s) [noise=true , address=145.79.227.45:4231 , piece_id=MICXNPBWQ6ZHWPL2CDB4KOH7GC3LSZJW2YXKVQJXCMENM6FGLZCQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.696615256 (584h19m33.567698291s) [noise=true , address=145.79.227.13:4804 , piece_id=QLGTAHKHVNKJDB3RAKRSJXYFLAQ5ZAW4JEDQENZ26KPPM62WUSXA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.700671430 (584h19m33.571754465s) [noise=true , address=145.79.227.7:4294 , piece_id=VVKZ5GNKZAMHXRD6GTU5EUW27QNDHSRNC6W4Y6C6XX5SGS6GTONQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.716500491 (584h19m33.587583536s) [noise=true , address=145.79.227.17:4972 , piece_id=S27WMVQVEQWYOFPVMWCPWEZN5YX7MI6JQSX56W3Z3RI3TX5T7G5A, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.723906112 (584h19m33.594989157s) [noise=true , address=145.79.227.47:4267 , piece_id=FBNNU4PX5P4DFQU5YC53GE353URBAYAXS5RQGOOUI6KOVRKNMSHA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.863810019 (584h19m33.734893054s) [noise=true , address=77.90.31.10:4518 , piece_id=RFFICLSMGOV5XHHHKDCO4TG7RAH3D4AYZCPBP65JPCIAD7VL2EUA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.887264209 (584h19m33.758347244s) [noise=true , address=77.90.31.14:4661 , piece_id=CIGRDFSNKNEMGJI5GTLLJ22Y3TR3QWSTQWLAVKPPHOYFOUZO6F6Q, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.970123234 (584h19m33.841206269s) [noise=true , address=45.91.116.12:4225 , piece_id=2X24WJOLZZMWGH7RHZHW76XOJXQVTCG47DGSBNDN7TV5TP4GE4CA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.970145396 (584h19m33.841228441s) [seg_pos={0 0}, upload=512232]: Segment reached optimal threshold of 13 pieces. Dec 29 23:48:33.970410942 (584h19m33.841493977s) [noise=true , address=145.79.227.5:4223 , piece_id=4QNWFLOW344XAN4QAFUQ7QOY2N764CB5HDY4Z52VL4HDMMNCKTIA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:52288->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:52288->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52288->145.79.227.5:4223: use of closed network connection Dec 29 23:48:33.970420540 (584h19m33.841503575s) [noise=true , address=145.79.227.5:4223 , piece_id=4QNWFLOW344XAN4QAFUQ7QOY2N764CB5HDY4Z52VL4HDMMNCKTIA, seg_pos={0 0}, upload=512232]: Long tail cancellation due to optimal threshold Dec 29 23:48:33.970494230 (584h19m33.841577265s) [noise=true , address=145.79.227.30:4762 , piece_id=GR5MLPLWVBYQABLPQPUTMMUHIQQIZ32UOUTRMKF6JNG4NTPV52GA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 23:48:33.970509861 (584h19m33.841592906s) [noise=true , address=145.79.227.30:4762 , piece_id=GR5MLPLWVBYQABLPQPUTMMUHIQQIZ32UOUTRMKF6JNG4NTPV52GA, seg_pos={0 0}, upload=512232]: Long tail cancellation due to optimal threshold Dec 29 23:48:33.970640348 (584h19m33.841723384s) [noise=true , address=45.91.116.6:4974 , piece_id=Z6YVJSGV5MTKIM5C5USKETVRY6UQOIRWSPBPHTHN37X64PAWHZAA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 23:48:33.970645799 (584h19m33.841728844s) [noise=true , address=45.91.116.6:4974 , piece_id=Z6YVJSGV5MTKIM5C5USKETVRY6UQOIRWSPBPHTHN37X64PAWHZAA, seg_pos={0 0}, upload=512232]: Long tail cancellation due to optimal threshold Dec 29 23:48:33.970667961 (584h19m33.841750996s) [seg_pos={0 0}, upload=512232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:48:33.970741080 (584h19m33.841824115s) [upload=512232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:48:33.970745709 (584h19m33.841828744s) [upload=512232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:48:34.278829755 (584h19m34.14991279s) [upload=512232]: Done uploading segments... Dec 29 23:48:34.278849203 (584h19m34.149932248s) [upload=512232]: Upload finished. err: <nil> Duration: 24.636636141s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.470008108s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-29 23:49:56 UTC) orbiter warning: context deadline exceeded Duration: 3.470032596s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X..X..X....X.....X......XX...X...X..XXX............XXX.] duration: 15.876857206s trace: https://jaeger.dev.storj.io/trace/18fd4c88bd813274 FAIL (2025-12-29 23:49:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :48:31.441307553 (584h19m31.312390588s) [upload=512232]: Starting upload Dec 29 23:48:31.441363339 (584h19m31.312446374s) [upload=512232]: Done starting upload Dec 29 23:48:31.441492194 (584h19m31.312575249s) [upload=512232]: Uploading segments... Dec 29 23:48:31.441658791 (584h19m31.312741836s) [upload=512232]: Got next segment. Inline: false Dec 29 23:48:31.441708046 (584h19m31.312791081s) [upload=512232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:48:31.441713927 (584h19m31.312796972s) [upload=512232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:48:31.691578184 (584h19m31.562661239s) [seg_pos={0 0}, upload=512232]: Begin upload segment... Dec 29 23:48:31.691637777 (584h19m31.562720812s) [seg_pos={0 0}, upload=512232]: Done begin upload segment. Dec 29 23:48:31.691688784 (584h19m31.562771819s) [noise=true , address=145.79.227.6:4258 , piece_id=OYJTQUX5FQQ6DTWKBU6BFQ3X5USCH3GBOUVDAXHRCZBHTILN2ZDQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.691700456 (584h19m31.562783501s) [noise=true , address=145.79.227.47:4267 , piece_id=FBNNU4PX5P4DFQU5YC53GE353URBAYAXS5RQGOOUI6KOVRKNMSHA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.691814223 (584h19m31.562897268s) [noise=true , address=77.90.31.14:4661 , piece_id=CIGRDFSNKNEMGJI5GTLLJ22Y3TR3QWSTQWLAVKPPHOYFOUZO6F6Q, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.692637470 (584h19m31.563720515s) [noise=true , address=145.79.227.45:4231 , piece_id=MICXNPBWQ6ZHWPL2CDB4KOH7GC3LSZJW2YXKVQJXCMENM6FGLZCQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.692653541 (584h19m31.563736586s) [noise=true , address=45.91.116.12:4225 , piece_id=2X24WJOLZZMWGH7RHZHW76XOJXQVTCG47DGSBNDN7TV5TP4GE4CA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.692704107 (584h19m31.563787162s) [noise=true , address=145.79.227.13:4804 , piece_id=QLGTAHKHVNKJDB3RAKRSJXYFLAQ5ZAW4JEDQENZ26KPPM62WUSXA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.693338315 (584h19m31.56442135s) [noise=true , address=145.79.227.11:4732 , piece_id=AXLV6RGTC3OWYE3AUZSOPYF6R62GTA27KTCOYWEOPHYXMJQPW6KQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.693758996 (584h19m31.564842031s) [noise=true , address=145.79.227.26:4653 , piece_id=II3A7L3PBV45OEZIPXTVW42LOJXXPZUKF6NHBPJ2WZLZLYCEHFZQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.694283244 (584h19m31.565366289s) [noise=true , address=45.91.116.6:4974 , piece_id=Z6YVJSGV5MTKIM5C5USKETVRY6UQOIRWSPBPHTHN37X64PAWHZAA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.694694106 (584h19m31.565777141s) [noise=true , address=145.79.227.30:4762 , piece_id=GR5MLPLWVBYQABLPQPUTMMUHIQQIZ32UOUTRMKF6JNG4NTPV52GA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.694821569 (584h19m31.565904604s) [noise=true , address=145.79.227.17:4972 , piece_id=S27WMVQVEQWYOFPVMWCPWEZN5YX7MI6JQSX56W3Z3RI3TX5T7G5A, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.695283439 (584h19m31.566366474s) [noise=true , address=145.79.227.5:4223 , piece_id=4QNWFLOW344XAN4QAFUQ7QOY2N764CB5HDY4Z52VL4HDMMNCKTIA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.695713587 (584h19m31.566796622s) [noise=true , address=77.90.31.10:4518 , piece_id=RFFICLSMGOV5XHHHKDCO4TG7RAH3D4AYZCPBP65JPCIAD7VL2EUA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.695998771 (584h19m31.567081816s) [noise=true , address=145.79.227.7:4294 , piece_id=VVKZ5GNKZAMHXRD6GTU5EUW27QNDHSRNC6W4Y6C6XX5SGS6GTONQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.696453337 (584h19m31.567536382s) [noise=true , address=145.79.227.3:4149 , piece_id=UREXLSG3A4XDPSH6TMFTVFTYBMI6EAMK2OXFLLG6TONP7OWXC72Q, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.697923306 (584h19m31.569006341s) [noise=true , address=145.79.227.14:4839 , piece_id=64JYSB7F4SOWFAJD4LPBLGI3DI6W3SNNSVB4JDDINU3HUAEC73HA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:33.390825229 (584h19m33.261908264s) [upload=512232]: Next returned nil segment Dec 29 23:48:33.390837111 (584h19m33.261920146s) [upload=512232]: Waiting for error group managing segments... Dec 29 23:48:33.663774052 (584h19m33.534857087s) [noise=true , address=145.79.227.3:4149 , piece_id=UREXLSG3A4XDPSH6TMFTVFTYBMI6EAMK2OXFLLG6TONP7OWXC72Q, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.668648524 (584h19m33.539731559s) [noise=true , address=145.79.227.11:4732 , piece_id=AXLV6RGTC3OWYE3AUZSOPYF6R62GTA27KTCOYWEOPHYXMJQPW6KQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.673561750 (584h19m33.544644785s) [noise=true , address=145.79.227.26:4653 , piece_id=II3A7L3PBV45OEZIPXTVW42LOJXXPZUKF6NHBPJ2WZLZLYCEHFZQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.673865738 (584h19m33.544948813s) [noise=true , address=145.79.227.14:4839 , piece_id=64JYSB7F4SOWFAJD4LPBLGI3DI6W3SNNSVB4JDDINU3HUAEC73HA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.677295810 (584h19m33.548378845s) [noise=true , address=145.79.227.6:4258 , piece_id=OYJTQUX5FQQ6DTWKBU6BFQ3X5USCH3GBOUVDAXHRCZBHTILN2ZDQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.684755484 (584h19m33.555838519s) [noise=true , address=145.79.227.45:4231 , piece_id=MICXNPBWQ6ZHWPL2CDB4KOH7GC3LSZJW2YXKVQJXCMENM6FGLZCQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.696615256 (584h19m33.567698291s) [noise=true , address=145.79.227.13:4804 , piece_id=QLGTAHKHVNKJDB3RAKRSJXYFLAQ5ZAW4JEDQENZ26KPPM62WUSXA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.700671430 (584h19m33.571754465s) [noise=true , address=145.79.227.7:4294 , piece_id=VVKZ5GNKZAMHXRD6GTU5EUW27QNDHSRNC6W4Y6C6XX5SGS6GTONQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.716500491 (584h19m33.587583536s) [noise=true , address=145.79.227.17:4972 , piece_id=S27WMVQVEQWYOFPVMWCPWEZN5YX7MI6JQSX56W3Z3RI3TX5T7G5A, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.723906112 (584h19m33.594989157s) [noise=true , address=145.79.227.47:4267 , piece_id=FBNNU4PX5P4DFQU5YC53GE353URBAYAXS5RQGOOUI6KOVRKNMSHA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.863810019 (584h19m33.734893054s) [noise=true , address=77.90.31.10:4518 , piece_id=RFFICLSMGOV5XHHHKDCO4TG7RAH3D4AYZCPBP65JPCIAD7VL2EUA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.887264209 (584h19m33.758347244s) [noise=true , address=77.90.31.14:4661 , piece_id=CIGRDFSNKNEMGJI5GTLLJ22Y3TR3QWSTQWLAVKPPHOYFOUZO6F6Q, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.970123234 (584h19m33.841206269s) [noise=true , address=45.91.116.12:4225 , piece_id=2X24WJOLZZMWGH7RHZHW76XOJXQVTCG47DGSBNDN7TV5TP4GE4CA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.970145396 (584h19m33.841228441s) [seg_pos={0 0}, upload=512232]: Segment reached optimal threshold of 13 pieces. Dec 29 23:48:33.970410942 (584h19m33.841493977s) [noise=true , address=145.79.227.5:4223 , piece_id=4QNWFLOW344XAN4QAFUQ7QOY2N764CB5HDY4Z52VL4HDMMNCKTIA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:52288->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:52288->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52288->145.79.227.5:4223: use of closed network connection Dec 29 23:48:33.970420540 (584h19m33.841503575s) [noise=true , address=145.79.227.5:4223 , piece_id=4QNWFLOW344XAN4QAFUQ7QOY2N764CB5HDY4Z52VL4HDMMNCKTIA, seg_pos={0 0}, upload=512232]: Long tail cancellation due to optimal threshold Dec 29 23:48:33.970494230 (584h19m33.841577265s) [noise=true , address=145.79.227.30:4762 , piece_id=GR5MLPLWVBYQABLPQPUTMMUHIQQIZ32UOUTRMKF6JNG4NTPV52GA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 23:48:33.970509861 (584h19m33.841592906s) [noise=true , address=145.79.227.30:4762 , piece_id=GR5MLPLWVBYQABLPQPUTMMUHIQQIZ32UOUTRMKF6JNG4NTPV52GA, seg_pos={0 0}, upload=512232]: Long tail cancellation due to optimal threshold Dec 29 23:48:33.970640348 (584h19m33.841723384s) [noise=true , address=45.91.116.6:4974 , piece_id=Z6YVJSGV5MTKIM5C5USKETVRY6UQOIRWSPBPHTHN37X64PAWHZAA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 23:48:33.970645799 (584h19m33.841728844s) [noise=true , address=45.91.116.6:4974 , piece_id=Z6YVJSGV5MTKIM5C5USKETVRY6UQOIRWSPBPHTHN37X64PAWHZAA, seg_pos={0 0}, upload=512232]: Long tail cancellation due to optimal threshold Dec 29 23:48:33.970667961 (584h19m33.841750996s) [seg_pos={0 0}, upload=512232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:48:33.970741080 (584h19m33.841824115s) [upload=512232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:48:33.970745709 (584h19m33.841828744s) [upload=512232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:48:34.278829755 (584h19m34.14991279s) [upload=512232]: Done uploading segments... Dec 29 23:48:34.278849203 (584h19m34.149932248s) [upload=512232]: Upload finished. err: <nil> Duration: 15.876876353s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.346224335s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-29 23:49:06 UTC) orbiter warning: context deadline exceeded Duration: 3.346260042s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......X..X..X....X.....X......XX...X...X..XXX............XXX] duration: 30.003397975s trace: https://jaeger.dev.storj.io/trace/18fd4c88bd813274 FAIL (2025-12-29 23:49:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :48:31.441307553 (584h19m31.312390588s) [upload=512232]: Starting upload Dec 29 23:48:31.441363339 (584h19m31.312446374s) [upload=512232]: Done starting upload Dec 29 23:48:31.441492194 (584h19m31.312575249s) [upload=512232]: Uploading segments... Dec 29 23:48:31.441658791 (584h19m31.312741836s) [upload=512232]: Got next segment. Inline: false Dec 29 23:48:31.441708046 (584h19m31.312791081s) [upload=512232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:48:31.441713927 (584h19m31.312796972s) [upload=512232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:48:31.691578184 (584h19m31.562661239s) [seg_pos={0 0}, upload=512232]: Begin upload segment... Dec 29 23:48:31.691637777 (584h19m31.562720812s) [seg_pos={0 0}, upload=512232]: Done begin upload segment. Dec 29 23:48:31.691688784 (584h19m31.562771819s) [noise=true , address=145.79.227.6:4258 , piece_id=OYJTQUX5FQQ6DTWKBU6BFQ3X5USCH3GBOUVDAXHRCZBHTILN2ZDQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.691700456 (584h19m31.562783501s) [noise=true , address=145.79.227.47:4267 , piece_id=FBNNU4PX5P4DFQU5YC53GE353URBAYAXS5RQGOOUI6KOVRKNMSHA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.691814223 (584h19m31.562897268s) [noise=true , address=77.90.31.14:4661 , piece_id=CIGRDFSNKNEMGJI5GTLLJ22Y3TR3QWSTQWLAVKPPHOYFOUZO6F6Q, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.692637470 (584h19m31.563720515s) [noise=true , address=145.79.227.45:4231 , piece_id=MICXNPBWQ6ZHWPL2CDB4KOH7GC3LSZJW2YXKVQJXCMENM6FGLZCQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.692653541 (584h19m31.563736586s) [noise=true , address=45.91.116.12:4225 , piece_id=2X24WJOLZZMWGH7RHZHW76XOJXQVTCG47DGSBNDN7TV5TP4GE4CA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.692704107 (584h19m31.563787162s) [noise=true , address=145.79.227.13:4804 , piece_id=QLGTAHKHVNKJDB3RAKRSJXYFLAQ5ZAW4JEDQENZ26KPPM62WUSXA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.693338315 (584h19m31.56442135s) [noise=true , address=145.79.227.11:4732 , piece_id=AXLV6RGTC3OWYE3AUZSOPYF6R62GTA27KTCOYWEOPHYXMJQPW6KQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.693758996 (584h19m31.564842031s) [noise=true , address=145.79.227.26:4653 , piece_id=II3A7L3PBV45OEZIPXTVW42LOJXXPZUKF6NHBPJ2WZLZLYCEHFZQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.694283244 (584h19m31.565366289s) [noise=true , address=45.91.116.6:4974 , piece_id=Z6YVJSGV5MTKIM5C5USKETVRY6UQOIRWSPBPHTHN37X64PAWHZAA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.694694106 (584h19m31.565777141s) [noise=true , address=145.79.227.30:4762 , piece_id=GR5MLPLWVBYQABLPQPUTMMUHIQQIZ32UOUTRMKF6JNG4NTPV52GA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.694821569 (584h19m31.565904604s) [noise=true , address=145.79.227.17:4972 , piece_id=S27WMVQVEQWYOFPVMWCPWEZN5YX7MI6JQSX56W3Z3RI3TX5T7G5A, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.695283439 (584h19m31.566366474s) [noise=true , address=145.79.227.5:4223 , piece_id=4QNWFLOW344XAN4QAFUQ7QOY2N764CB5HDY4Z52VL4HDMMNCKTIA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.695713587 (584h19m31.566796622s) [noise=true , address=77.90.31.10:4518 , piece_id=RFFICLSMGOV5XHHHKDCO4TG7RAH3D4AYZCPBP65JPCIAD7VL2EUA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.695998771 (584h19m31.567081816s) [noise=true , address=145.79.227.7:4294 , piece_id=VVKZ5GNKZAMHXRD6GTU5EUW27QNDHSRNC6W4Y6C6XX5SGS6GTONQ, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.696453337 (584h19m31.567536382s) [noise=true , address=145.79.227.3:4149 , piece_id=UREXLSG3A4XDPSH6TMFTVFTYBMI6EAMK2OXFLLG6TONP7OWXC72Q, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:31.697923306 (584h19m31.569006341s) [noise=true , address=145.79.227.14:4839 , piece_id=64JYSB7F4SOWFAJD4LPBLGI3DI6W3SNNSVB4JDDINU3HUAEC73HA, seg_pos={0 0}, upload=512232]: Uploading piece... Dec 29 23:48:33.390825229 (584h19m33.261908264s) [upload=512232]: Next returned nil segment Dec 29 23:48:33.390837111 (584h19m33.261920146s) [upload=512232]: Waiting for error group managing segments... Dec 29 23:48:33.663774052 (584h19m33.534857087s) [noise=true , address=145.79.227.3:4149 , piece_id=UREXLSG3A4XDPSH6TMFTVFTYBMI6EAMK2OXFLLG6TONP7OWXC72Q, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.668648524 (584h19m33.539731559s) [noise=true , address=145.79.227.11:4732 , piece_id=AXLV6RGTC3OWYE3AUZSOPYF6R62GTA27KTCOYWEOPHYXMJQPW6KQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.673561750 (584h19m33.544644785s) [noise=true , address=145.79.227.26:4653 , piece_id=II3A7L3PBV45OEZIPXTVW42LOJXXPZUKF6NHBPJ2WZLZLYCEHFZQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.673865738 (584h19m33.544948813s) [noise=true , address=145.79.227.14:4839 , piece_id=64JYSB7F4SOWFAJD4LPBLGI3DI6W3SNNSVB4JDDINU3HUAEC73HA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.677295810 (584h19m33.548378845s) [noise=true , address=145.79.227.6:4258 , piece_id=OYJTQUX5FQQ6DTWKBU6BFQ3X5USCH3GBOUVDAXHRCZBHTILN2ZDQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.684755484 (584h19m33.555838519s) [noise=true , address=145.79.227.45:4231 , piece_id=MICXNPBWQ6ZHWPL2CDB4KOH7GC3LSZJW2YXKVQJXCMENM6FGLZCQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.696615256 (584h19m33.567698291s) [noise=true , address=145.79.227.13:4804 , piece_id=QLGTAHKHVNKJDB3RAKRSJXYFLAQ5ZAW4JEDQENZ26KPPM62WUSXA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.700671430 (584h19m33.571754465s) [noise=true , address=145.79.227.7:4294 , piece_id=VVKZ5GNKZAMHXRD6GTU5EUW27QNDHSRNC6W4Y6C6XX5SGS6GTONQ, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.716500491 (584h19m33.587583536s) [noise=true , address=145.79.227.17:4972 , piece_id=S27WMVQVEQWYOFPVMWCPWEZN5YX7MI6JQSX56W3Z3RI3TX5T7G5A, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.723906112 (584h19m33.594989157s) [noise=true , address=145.79.227.47:4267 , piece_id=FBNNU4PX5P4DFQU5YC53GE353URBAYAXS5RQGOOUI6KOVRKNMSHA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.863810019 (584h19m33.734893054s) [noise=true , address=77.90.31.10:4518 , piece_id=RFFICLSMGOV5XHHHKDCO4TG7RAH3D4AYZCPBP65JPCIAD7VL2EUA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.887264209 (584h19m33.758347244s) [noise=true , address=77.90.31.14:4661 , piece_id=CIGRDFSNKNEMGJI5GTLLJ22Y3TR3QWSTQWLAVKPPHOYFOUZO6F6Q, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.970123234 (584h19m33.841206269s) [noise=true , address=45.91.116.12:4225 , piece_id=2X24WJOLZZMWGH7RHZHW76XOJXQVTCG47DGSBNDN7TV5TP4GE4CA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: <nil> Dec 29 23:48:33.970145396 (584h19m33.841228441s) [seg_pos={0 0}, upload=512232]: Segment reached optimal threshold of 13 pieces. Dec 29 23:48:33.970410942 (584h19m33.841493977s) [noise=true , address=145.79.227.5:4223 , piece_id=4QNWFLOW344XAN4QAFUQ7QOY2N764CB5HDY4Z52VL4HDMMNCKTIA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:52288->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:52288->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52288->145.79.227.5:4223: use of closed network connection Dec 29 23:48:33.970420540 (584h19m33.841503575s) [noise=true , address=145.79.227.5:4223 , piece_id=4QNWFLOW344XAN4QAFUQ7QOY2N764CB5HDY4Z52VL4HDMMNCKTIA, seg_pos={0 0}, upload=512232]: Long tail cancellation due to optimal threshold Dec 29 23:48:33.970494230 (584h19m33.841577265s) [noise=true , address=145.79.227.30:4762 , piece_id=GR5MLPLWVBYQABLPQPUTMMUHIQQIZ32UOUTRMKF6JNG4NTPV52GA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 23:48:33.970509861 (584h19m33.841592906s) [noise=true , address=145.79.227.30:4762 , piece_id=GR5MLPLWVBYQABLPQPUTMMUHIQQIZ32UOUTRMKF6JNG4NTPV52GA, seg_pos={0 0}, upload=512232]: Long tail cancellation due to optimal threshold Dec 29 23:48:33.970640348 (584h19m33.841723384s) [noise=true , address=45.91.116.6:4974 , piece_id=Z6YVJSGV5MTKIM5C5USKETVRY6UQOIRWSPBPHTHN37X64PAWHZAA, seg_pos={0 0}, upload=512232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 23:48:33.970645799 (584h19m33.841728844s) [noise=true , address=45.91.116.6:4974 , piece_id=Z6YVJSGV5MTKIM5C5USKETVRY6UQOIRWSPBPHTHN37X64PAWHZAA, seg_pos={0 0}, upload=512232]: Long tail cancellation due to optimal threshold Dec 29 23:48:33.970667961 (584h19m33.841750996s) [seg_pos={0 0}, upload=512232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:48:33.970741080 (584h19m33.841824115s) [upload=512232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:48:33.970745709 (584h19m33.841828744s) [upload=512232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:48:34.278829755 (584h19m34.14991279s) [upload=512232]: Done uploading segments... Dec 29 23:48:34.278849203 (584h19m34.149932248s) [upload=512232]: Upload finished. err: <nil> Duration: 30.003421232s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.498718074s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-29 23:48:40 UTC) orbiter warning: context deadline exceeded Duration: 3.498746387s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......X..X..X....X.....X......XX...X...X..XXX............XX] duration: 30.00318898s trace: https://jaeger.dev.storj.io/trace/3374c335805e9c61 FAIL (2025-12-29 23:47:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ments... Dec 29 23:47:23.537248039 (584h18m23.408331074s) [upload=512216]: Got next segment. Inline: false Dec 29 23:47:23.537274870 (584h18m23.408357905s) [upload=512216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:47:23.537278727 (584h18m23.408361762s) [upload=512216]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:47:23.786821701 (584h18m23.657904736s) [seg_pos={0 0}, upload=512216]: Begin upload segment... Dec 29 23:47:23.786863179 (584h18m23.657946224s) [seg_pos={0 0}, upload=512216]: Done begin upload segment. Dec 29 23:47:23.786879331 (584h18m23.657962366s) [noise=true , address=145.79.227.18:4006 , piece_id=RESVB6WXKSF5ZVDIPMRJ4HAIRGXITZK3C3M4IALIKGSMQSL4FK2Q, seg_pos={0 0}, upload=512216]: Uploading piece... Dec 29 23:47:23.786891804 (584h18m23.657974839s) [noise=true , address=77.90.31.21:4877 , piece_id=N7OACIAC6SL54R3EAP4EYJA42DRLGJGEJ7QV3ZOBM66SKATAD7JQ, seg_pos={0 0}, upload=512216]: Uploading piece... Dec 29 23:47:23.786975723 (584h18m23.658058758s) [noise=true , address=77.90.31.10:4517 , piece_id=XRL7EXHLIPY2RQUZ3X6VCTVQWZZYVDBJRNMARUZIZXESBSLQ6GFQ, seg_pos={0 0}, upload=512216]: Uploading piece... Dec 29 23:47:23.787769225 (584h18m23.65885226s) [noise=true , address=145.79.227.52:4446 , piece_id=YM37ZVR3KN3VWGTCR2LTEPYPOAIXH2SFBT7L4Q3EH7ODXBOUGA3Q, seg_pos={0 0}, upload=512216]: Uploading piece... Dec 29 23:47:23.788100185 (584h18m23.65918323s) [noise=true , address=145.79.227.7:4295 , piece_id=CX4KGAZ6MVHVEKFERCHPMQNF5RMEUVUR3HTQ77J3WXBUMG4A4FOA, seg_pos={0 0}, upload=512216]: Uploading piece... Dec 29 23:47:23.788379026 (584h18m23.659462061s) [noise=true , address=77.90.31.12:4590 , piece_id=ZIDQMZG2VLEW2QGZMZSWONL5CGWEFL3BGWZMYWAUHQVEVH3DIGOQ, seg_pos={0 0}, upload=512216]: Uploading piece... Dec 29 23:47:23.788695238 (584h18m23.659778273s) [noise=true , address=145.79.227.46:4601 , piece_id=LXOM4IP6MCFBUPRASSIAF3BFV6JI5PWCSXAT52HD7H3WO6XRNQDA, seg_pos={0 0}, upload=512216]: Uploading piece... Dec 29 23:47:23.788750884 (584h18m23.659833919s) [noise=true , address=45.91.116.14:4190 , piece_id=3UX6RIOT7MFE3C2XFHBMCGX7REFTJTQSB5WUJJIZJ6FV274HCM5Q, seg_pos={0 0}, upload=512216]: Uploading piece... Dec 29 23:47:23.789853053 (584h18m23.660936098s) [noise=true , address=77.90.31.13:4626 , piece_id=D3HULBMJYFPFZVWU7E3AY4G3SSDJ3DPFBEFBQZQICLYCDDFWALWA, seg_pos={0 0}, upload=512216]: Uploading piece... Dec 29 23:47:23.790543738 (584h18m23.661626773s) [noise=true , address=45.91.116.9:4263 , piece_id=EIEUR5TAR7E6CPQHPFAM2OGNKARDOPECXS5RTXJHD4FMZYGQW3BA, seg_pos={0 0}, upload=512216]: Uploading piece... Dec 29 23:47:23.791611752 (584h18m23.662694787s) [noise=true , address=145.79.227.2:4114 , piece_id=2XJF66V2EHAQIVAJWEVK6VDCHM4NCPNUL6VWEEK4UB5RXN7AMGUQ, seg_pos={0 0}, upload=512216]: Uploading piece... Dec 29 23:47:23.793148959 (584h18m23.664231994s) [noise=true , address=145.79.227.39:4014 , piece_id=ATCPNHTV4UPEEAP6PS4YZVPKO2CN2LQODYFDZCKGFPLMADGRPQZQ, seg_pos={0 0}, upload=512216]: Uploading piece... Dec 29 23:47:23.794522474 (584h18m23.665605509s) [noise=true , address=145.79.227.37:4567 , piece_id=VRFR5MYFZHT3YX2A3HGPTTX4G2IWSYMLERJYT5REGAU6VGAXNNGQ, seg_pos={0 0}, upload=512216]: Uploading piece... Dec 29 23:47:23.795645302 (584h18m23.666728337s) [noise=true , address=145.79.227.11:4731 , piece_id=4VOFQJ7O54VVRKDNRA3UEAM6JP2MBMYLWDGYBRXS5QFOUZOPINMQ, seg_pos={0 0}, upload=512216]: Uploading piece... Dec 29 23:47:23.796800272 (584h18m23.667883307s) [noise=true , address=145.79.227.6:4258 , piece_id=XX6FBGD4HDSASHHRIH2GE5UYIWQ3FD4UGRHX2FCPGEJ3HHF234XQ, seg_pos={0 0}, upload=512216]: Uploading piece... Dec 29 23:47:23.798005256 (584h18m23.669088291s) [noise=true , address=145.79.227.51:4409 , piece_id=EZ2DJHYNQ73OKHAAAT2VJJ3Z6E64LKQLKDFGYOEHX7EE76IMOQQA, seg_pos={0 0}, upload=512216]: Uploading piece... Dec 29 23:47:25.455621338 (584h18m25.326704373s) [upload=512216]: Next returned nil segment Dec 29 23:47:25.455634353 (584h18m25.326717389s) [upload=512216]: Waiting for error group managing segments... Dec 29 23:47:25.767302173 (584h18m25.638385208s) [noise=true , address=145.79.227.7:4295 , piece_id=CX4KGAZ6MVHVEKFERCHPMQNF5RMEUVUR3HTQ77J3WXBUMG4A4FOA, seg_pos={0 0}, upload=512216]: Done uploading piece. err: <nil> Dec 29 23:47:25.773614682 (584h18m25.644697717s) [noise=true , address=145.79.227.11:4731 , piece_id=4VOFQJ7O54VVRKDNRA3UEAM6JP2MBMYLWDGYBRXS5QFOUZOPINMQ, seg_pos={0 0}, upload=512216]: Done uploading piece. err: <nil> Dec 29 23:47:25.775632415 (584h18m25.64671546s) [noise=true , address=145.79.227.52:4446 , piece_id=YM37ZVR3KN3VWGTCR2LTEPYPOAIXH2SFBT7L4Q3EH7ODXBOUGA3Q, seg_pos={0 0}, upload=512216]: Done uploading piece. err: <nil> Dec 29 23:47:25.777686796 (584h18m25.648769831s) [noise=true , address=145.79.227.18:4006 , piece_id=RESVB6WXKSF5ZVDIPMRJ4HAIRGXITZK3C3M4IALIKGSMQSL4FK2Q, seg_pos={0 0}, upload=512216]: Done uploading piece. err: <nil> Dec 29 23:47:25.790468655 (584h18m25.6615517s) [noise=true , address=145.79.227.51:4409 , piece_id=EZ2DJHYNQ73OKHAAAT2VJJ3Z6E64LKQLKDFGYOEHX7EE76IMOQQA, seg_pos={0 0}, upload=512216]: Done uploading piece. err: <nil> Dec 29 23:47:25.793409625 (584h18m25.66449266s) [noise=true , address=145.79.227.6:4258 , piece_id=XX6FBGD4HDSASHHRIH2GE5UYIWQ3FD4UGRHX2FCPGEJ3HHF234XQ, seg_pos={0 0}, upload=512216]: Done uploading piece. err: <nil> Dec 29 23:47:25.795743149 (584h18m25.666826184s) [noise=true , address=145.79.227.37:4567 , piece_id=VRFR5MYFZHT3YX2A3HGPTTX4G2IWSYMLERJYT5REGAU6VGAXNNGQ, seg_pos={0 0}, upload=512216]: Done uploading piece. err: <nil> Dec 29 23:47:25.807758869 (584h18m25.678841904s) [noise=true , address=145.79.227.39:4014 , piece_id=ATCPNHTV4UPEEAP6PS4YZVPKO2CN2LQODYFDZCKGFPLMADGRPQZQ, seg_pos={0 0}, upload=512216]: Done uploading piece. err: <nil> Dec 29 23:47:25.813681966 (584h18m25.684765001s) [noise=true , address=145.79.227.2:4114 , piece_id=2XJF66V2EHAQIVAJWEVK6VDCHM4NCPNUL6VWEEK4UB5RXN7AMGUQ, seg_pos={0 0}, upload=512216]: Done uploading piece. err: <nil> Dec 29 23:47:25.935776431 (584h18m25.806859466s) [noise=true , address=77.90.31.10:4517 , piece_id=XRL7EXHLIPY2RQUZ3X6VCTVQWZZYVDBJRNMARUZIZXESBSLQ6GFQ, seg_pos={0 0}, upload=512216]: Done uploading piece. err: <nil> Dec 29 23:47:25.988728089 (584h18m25.859811124s) [noise=true , address=77.90.31.13:4626 , piece_id=D3HULBMJYFPFZVWU7E3AY4G3SSDJ3DPFBEFBQZQICLYCDDFWALWA, seg_pos={0 0}, upload=512216]: Done uploading piece. err: <nil> Dec 29 23:47:26.004199328 (584h18m25.875282373s) [noise=true , address=77.90.31.21:4877 , piece_id=N7OACIAC6SL54R3EAP4EYJA42DRLGJGEJ7QV3ZOBM66SKATAD7JQ, seg_pos={0 0}, upload=512216]: Done uploading piece. err: <nil> Dec 29 23:47:26.041903533 (584h18m25.912986568s) [noise=true , address=45.91.116.14:4190 , piece_id=3UX6RIOT7MFE3C2XFHBMCGX7REFTJTQSB5WUJJIZJ6FV274HCM5Q, seg_pos={0 0}, upload=512216]: Done uploading piece. err: <nil> Dec 29 23:47:26.041930593 (584h18m25.913013628s) [seg_pos={0 0}, upload=512216]: Segment reached optimal threshold of 13 pieces. Dec 29 23:47:26.042084036 (584h18m25.913167071s) [noise=true , address=77.90.31.12:4590 , piece_id=ZIDQMZG2VLEW2QGZMZSWONL5CGWEFL3BGWZMYWAUHQVEVH3DIGOQ, seg_pos={0 0}, upload=512216]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 29 23:47:26.042091821 (584h18m25.913174866s) [noise=true , address=77.90.31.12:4590 , piece_id=ZIDQMZG2VLEW2QGZMZSWONL5CGWEFL3BGWZMYWAUHQVEVH3DIGOQ, seg_pos={0 0}, upload=512216]: Long tail cancellation due to optimal threshold Dec 29 23:47:26.042273837 (584h18m25.913356872s) [noise=true , address=45.91.116.9:4263 , piece_id=EIEUR5TAR7E6CPQHPFAM2OGNKARDOPECXS5RTXJHD4FMZYGQW3BA, seg_pos={0 0}, upload=512216]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:37276->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:37276->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37276->45.91.116.9:4263: use of closed network connection Dec 29 23:47:26.042289227 (584h18m25.913372262s) [noise=true , address=45.91.116.9:4263 , piece_id=EIEUR5TAR7E6CPQHPFAM2OGNKARDOPECXS5RTXJHD4FMZYGQW3BA, seg_pos={0 0}, upload=512216]: Long tail cancellation due to optimal threshold Dec 29 23:47:26.042355764 (584h18m25.913438809s) [noise=true , address=145.79.227.46:4601 , piece_id=LXOM4IP6MCFBUPRASSIAF3BFV6JI5PWCSXAT52HD7H3WO6XRNQDA, seg_pos={0 0}, upload=512216]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: write tcp 188.166.225.170:49678->145.79.227.46:4601: use of closed network connection; write tcp 188.166.225.170:49678->145.79.227.46:4601: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49678->145.79.227.46:4601: use of closed network connection Dec 29 23:47:26.042363378 (584h18m25.913446413s) [noise=true , address=145.79.227.46:4601 , piece_id=LXOM4IP6MCFBUPRASSIAF3BFV6JI5PWCSXAT52HD7H3WO6XRNQDA, seg_pos={0 0}, upload=512216]: Long tail cancellation due to optimal threshold Dec 29 23:47:26.042376974 (584h18m25.913460009s) [seg_pos={0 0}, upload=512216]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:47:26.042565754 (584h18m25.913648789s) [upload=512216]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:47:26.042572687 (584h18m25.913655722s) [upload=512216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:47:26.333939439 (584h18m26.205022504s) [upload=512216]: Done uploading segments... Dec 29 23:47:26.333963134 (584h18m26.205046169s) [upload=512216]: Upload finished. err: <nil> Duration: 30.003198087s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001040367s trace: https://jaeger.dev.storj.io/trace/425165442410e0ad FAIL (2025-12-29 23:47:48 UTC) orbiter warning: context deadline exceeded Duration: 5.001062049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........X..X..X....X.....X......XX...X...X..XXX............X] duration: 30.001806421s trace: https://jaeger.dev.storj.io/trace/45a7e9efd4cfc9a FAIL (2025-12-29 23:47:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 23:46:53.304119038 (584h17m53.175202073s) [upload=512209]: Starting upload Dec 29 23:46:53.304193640 (584h17m53.175276675s) [upload=512209]: Done starting upload Dec 29 23:46:53.304275996 (584h17m53.175359041s) [upload=512209]: Uploading segments... Dec 29 23:46:53.304371408 (584h17m53.175454443s) [upload=512209]: Got next segment. Inline: false Dec 29 23:46:53.304404792 (584h17m53.175487827s) [upload=512209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:46:53.304409351 (584h17m53.175492386s) [upload=512209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:46:53.549840035 (584h17m53.420923091s) [seg_pos={0 0}, upload=512209]: Begin upload segment... Dec 29 23:46:53.549911391 (584h17m53.420994436s) [seg_pos={0 0}, upload=512209]: Done begin upload segment. Dec 29 23:46:53.549949173 (584h17m53.421032218s) [noise=true , address=77.90.31.16:4735 , piece_id=4GTI4QT3KXLYOG6CVFM2FCTWZUF6P3BFXGHKYCVFRLXLSTXPK6RA, seg_pos={0 0}, upload=512209]: Uploading piece... Dec 29 23:46:53.549957400 (584h17m53.421040455s) [noise=true , address=145.79.227.12:4765 , piece_id=6I2BLGG37VM3SK46PZQWSLNBFDJPJDZG3JKYDYIIIRTYXPZU5YAQ, seg_pos={0 0}, upload=512209]: Uploading piece... Dec 29 23:46:53.550036019 (584h17m53.421119064s) [noise=true , address=145.79.227.4:4185 , piece_id=6IEH6M3EPRBRBHCDYCTBGQMLL273KHWER7HVNQEHE25PNIJR23BA, seg_pos={0 0}, upload=512209]: Uploading piece... Dec 29 23:46:53.550712697 (584h17m53.421795732s) [noise=true , address=145.79.227.14:4840 , piece_id=6TSFVIR2AWJ4FX7BYEKAAM2HEQ5HIYRG6WHE32JG5BMDW46DGWLQ, seg_pos={0 0}, upload=512209]: Uploading piece... Dec 29 23:46:53.551239791 (584h17m53.422322836s) [noise=true , address=77.90.31.19:4844 , piece_id=ZSVEALTRWUXJIE6VEUNZQAFUZ4J3WEYY7I22XP7LJ7B6PWC7WVCA, seg_pos={0 0}, upload=512209]: Uploading piece... Dec 29 23:46:53.551631187 (584h17m53.422714232s) [noise=true , address=145.79.227.15:4874 , piece_id=IGYAFBHLTVIMLSOMHQ7WIB7UBB2EOGOVCSW3VWXPNNRTZIYKIMUA, seg_pos={0 0}, upload=512209]: Uploading piece... Dec 29 23:46:53.552139674 (584h17m53.423222719s) [noise=true , address=145.79.227.7:4293 , piece_id=RK3P6ZLBXVBVJN3XMCGLS4XFOBUAFJZ43N25XZ5UL4FX6VB3CWQQ, seg_pos={0 0}, upload=512209]: Uploading piece... Dec 29 23:46:53.552278108 (584h17m53.423361163s) [noise=true , address=145.79.227.37:4566 , piece_id=QUCBPEJWR3QCBEB4T7MVETKLHK2SACSRTA4H26QMAHOFXST2VRNQ, seg_pos={0 0}, upload=512209]: Uploading piece... Dec 29 23:46:53.552531792 (584h17m53.423614837s) [noise=true , address=45.91.116.8:4047 , piece_id=QE7VDEEQQSZJMJZQ5PWDJ6HP4QRX3NTETDP5YN6SM5AQRAZC47HQ, seg_pos={0 0}, upload=512209]: Uploading piece... Dec 29 23:46:53.554212913 (584h17m53.425295958s) [noise=true , address=45.91.116.13:4153 , piece_id=UPPCGZ23Y2TKGXWH564JABM7IE6EENLICODDD2R2RDTQHSLC6RAA, seg_pos={0 0}, upload=512209]: Uploading piece... Dec 29 23:46:53.554641238 (584h17m53.425724283s) [noise=true , address=77.90.31.13:4626 , piece_id=SDGU7VZQMDNEML53Q23NCV3RZS4ELJFGRWMOX57KK5IEXFPJ5BQA, seg_pos={0 0}, upload=512209]: Uploading piece... Dec 29 23:46:53.555094591 (584h17m53.426177636s) [noise=true , address=145.79.227.50:4375 , piece_id=4CKCYFYU3UFWY4BH2NYFKK4DSAFJV7NZVGQKPGJGRGBZ7YA5NMSQ, seg_pos={0 0}, upload=512209]: Uploading piece... Dec 29 23:46:53.555567040 (584h17m53.426650085s) [noise=true , address=145.79.227.30:4762 , piece_id=NXV2MUEG3YPPYDONJTDS57AMMCAQEZGDTN3K66Z6MCO7K63TKAIQ, seg_pos={0 0}, upload=512209]: Uploading piece... Dec 29 23:46:53.556257856 (584h17m53.427340901s) [noise=true , address=145.79.227.6:4257 , piece_id=DQTIL3ZGD6BOPKKG2N72DTLBQJA3JPFJBTZHB7NMTLNPS63XD5EQ, seg_pos={0 0}, upload=512209]: Uploading piece... Dec 29 23:46:53.556778177 (584h17m53.427861222s) [noise=true , address=45.91.116.5:4298 , piece_id=MWPC6WJ2PEKAU7DNO6G25ZPKT4UWAZLFJZ2WIER66GPXWDYIIPXA, seg_pos={0 0}, upload=512209]: Uploading piece... Dec 29 23:46:53.558552345 (584h17m53.42963539s) [noise=true , address=145.79.227.45:4230 , piece_id=IWVQCOSR5CQ6GNHWUUIJRYNW6YPNRBL4CJ3UTX3S5SEZMTHEE44A, seg_pos={0 0}, upload=512209]: Uploading piece... Dec 29 23:46:55.246720416 (584h17m55.117803451s) [upload=512209]: Next returned nil segment Dec 29 23:46:55.246732288 (584h17m55.117815323s) [upload=512209]: Waiting for error group managing segments... Dec 29 23:46:55.525234055 (584h17m55.39631709s) [noise=true , address=145.79.227.45:4230 , piece_id=IWVQCOSR5CQ6GNHWUUIJRYNW6YPNRBL4CJ3UTX3S5SEZMTHEE44A, seg_pos={0 0}, upload=512209]: Done uploading piece. err: <nil> Dec 29 23:46:55.540485106 (584h17m55.411568151s) [noise=true , address=145.79.227.37:4566 , piece_id=QUCBPEJWR3QCBEB4T7MVETKLHK2SACSRTA4H26QMAHOFXST2VRNQ, seg_pos={0 0}, upload=512209]: Done uploading piece. err: <nil> Dec 29 23:46:55.541006068 (584h17m55.412089133s) [noise=true , address=145.79.227.14:4840 , piece_id=6TSFVIR2AWJ4FX7BYEKAAM2HEQ5HIYRG6WHE32JG5BMDW46DGWLQ, seg_pos={0 0}, upload=512209]: Done uploading piece. err: <nil> Dec 29 23:46:55.545742906 (584h17m55.416825941s) [noise=true , address=145.79.227.4:4185 , piece_id=6IEH6M3EPRBRBHCDYCTBGQMLL273KHWER7HVNQEHE25PNIJR23BA, seg_pos={0 0}, upload=512209]: Done uploading piece. err: <nil> Dec 29 23:46:55.546321609 (584h17m55.417404654s) [noise=true , address=145.79.227.6:4257 , piece_id=DQTIL3ZGD6BOPKKG2N72DTLBQJA3JPFJBTZHB7NMTLNPS63XD5EQ, seg_pos={0 0}, upload=512209]: Done uploading piece. err: <nil> Dec 29 23:46:55.546706331 (584h17m55.417789366s) [noise=true , address=145.79.227.15:4874 , piece_id=IGYAFBHLTVIMLSOMHQ7WIB7UBB2EOGOVCSW3VWXPNNRTZIYKIMUA, seg_pos={0 0}, upload=512209]: Done uploading piece. err: <nil> Dec 29 23:46:55.548180007 (584h17m55.419263052s) [noise=true , address=145.79.227.12:4765 , piece_id=6I2BLGG37VM3SK46PZQWSLNBFDJPJDZG3JKYDYIIIRTYXPZU5YAQ, seg_pos={0 0}, upload=512209]: Done uploading piece. err: <nil> Dec 29 23:46:55.552479124 (584h17m55.423562159s) [noise=true , address=145.79.227.7:4293 , piece_id=RK3P6ZLBXVBVJN3XMCGLS4XFOBUAFJZ43N25XZ5UL4FX6VB3CWQQ, seg_pos={0 0}, upload=512209]: Done uploading piece. err: <nil> Dec 29 23:46:55.557336983 (584h17m55.428420029s) [noise=true , address=145.79.227.50:4375 , piece_id=4CKCYFYU3UFWY4BH2NYFKK4DSAFJV7NZVGQKPGJGRGBZ7YA5NMSQ, seg_pos={0 0}, upload=512209]: Done uploading piece. err: <nil> Dec 29 23:46:55.872930017 (584h17m55.744013062s) [noise=true , address=77.90.31.19:4844 , piece_id=ZSVEALTRWUXJIE6VEUNZQAFUZ4J3WEYY7I22XP7LJ7B6PWC7WVCA, seg_pos={0 0}, upload=512209]: Done uploading piece. err: <nil> Dec 29 23:46:55.906129774 (584h17m55.77721281s) [noise=true , address=45.91.116.5:4298 , piece_id=MWPC6WJ2PEKAU7DNO6G25ZPKT4UWAZLFJZ2WIER66GPXWDYIIPXA, seg_pos={0 0}, upload=512209]: Done uploading piece. err: <nil> Dec 29 23:46:55.918930288 (584h17m55.790013333s) [noise=true , address=45.91.116.13:4153 , piece_id=UPPCGZ23Y2TKGXWH564JABM7IE6EENLICODDD2R2RDTQHSLC6RAA, seg_pos={0 0}, upload=512209]: Done uploading piece. err: <nil> Dec 29 23:46:56.095165676 (584h17m55.966248711s) [noise=true , address=45.91.116.8:4047 , piece_id=QE7VDEEQQSZJMJZQ5PWDJ6HP4QRX3NTETDP5YN6SM5AQRAZC47HQ, seg_pos={0 0}, upload=512209]: Done uploading piece. err: <nil> Dec 29 23:46:56.095186967 (584h17m55.966270032s) [seg_pos={0 0}, upload=512209]: Segment reached optimal threshold of 13 pieces. Dec 29 23:46:56.095432344 (584h17m55.966515389s) [noise=true , address=77.90.31.16:4735 , piece_id=4GTI4QT3KXLYOG6CVFM2FCTWZUF6P3BFXGHKYCVFRLXLSTXPK6RA, seg_pos={0 0}, upload=512209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 29 23:46:56.095444477 (584h17m55.966527512s) [noise=true , address=77.90.31.16:4735 , piece_id=4GTI4QT3KXLYOG6CVFM2FCTWZUF6P3BFXGHKYCVFRLXLSTXPK6RA, seg_pos={0 0}, upload=512209]: Long tail cancellation due to optimal threshold Dec 29 23:46:56.095534248 (584h17m55.966617283s) [noise=true , address=77.90.31.13:4626 , piece_id=SDGU7VZQMDNEML53Q23NCV3RZS4ELJFGRWMOX57KK5IEXFPJ5BQA, seg_pos={0 0}, upload=512209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:41628->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:41628->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41628->77.90.31.13:4626: use of closed network connection Dec 29 23:46:56.095557653 (584h17m55.966640688s) [noise=true , address=77.90.31.13:4626 , piece_id=SDGU7VZQMDNEML53Q23NCV3RZS4ELJFGRWMOX57KK5IEXFPJ5BQA, seg_pos={0 0}, upload=512209]: Long tail cancellation due to optimal threshold Dec 29 23:46:56.095694102 (584h17m55.966777137s) [noise=true , address=145.79.227.30:4762 , piece_id=NXV2MUEG3YPPYDONJTDS57AMMCAQEZGDTN3K66Z6MCO7K63TKAIQ, seg_pos={0 0}, upload=512209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 23:46:56.095699744 (584h17m55.966782779s) [noise=true , address=145.79.227.30:4762 , piece_id=NXV2MUEG3YPPYDONJTDS57AMMCAQEZGDTN3K66Z6MCO7K63TKAIQ, seg_pos={0 0}, upload=512209]: Long tail cancellation due to optimal threshold Dec 29 23:46:56.095716034 (584h17m55.966799069s) [seg_pos={0 0}, upload=512209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:46:56.095786278 (584h17m55.966869313s) [upload=512209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:46:56.095791748 (584h17m55.966874783s) [upload=512209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:46:56.389861690 (584h17m56.260944725s) [upload=512209]: Done uploading segments... Dec 29 23:46:56.389879593 (584h17m56.260962628s) [upload=512209]: Upload finished. err: <nil> Duration: 30.001825777s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.........X..X..X....X.....X......XX...X...X..XXX............] duration: 17.948470417s trace: https://jaeger.dev.storj.io/trace/2215ac2420cc0f18 FAIL (2025-12-29 23:46:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: GW5CGV5RHYMG6YQSFQDODMKSTQ5U66H6VDSI3EN7BRQE7AVT5KCA) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: CRF4HS5WWUVREG3L4MN5ZDSIXUSXJP7OCLODRSL4MD2ME2BZUWVQ) context deadline exceeded ======LOGS====== h7m26.358879078s) [upload=512056]: Uploading segments... Dec 29 23:36:26.487851669 (584h7m26.358934714s) [upload=512056]: Got next segment. Inline: false Dec 29 23:36:26.487880584 (584h7m26.358963629s) [upload=512056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:36:26.487884982 (584h7m26.358968017s) [upload=512056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:36:26.730337976 (584h7m26.601421011s) [seg_pos={0 0}, upload=512056]: Begin upload segment... Dec 29 23:36:26.730457603 (584h7m26.601540638s) [seg_pos={0 0}, upload=512056]: Done begin upload segment. Dec 29 23:36:26.730491598 (584h7m26.601574643s) [noise=true , address=77.90.31.14:4664 , piece_id=CRF4HS5WWUVREG3L4MN5ZDSIXUSXJP7OCLODRSL4MD2ME2BZUWVQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.730744679 (584h7m26.601827725s) [noise=true , address=145.79.227.46:4602 , piece_id=BSZRANNWJP2EZ2R74LXX4NSJXWHIEUTHMKIALZBW7ZFKUVXVQAKQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731249752 (584h7m26.602332797s) [noise=true , address=45.91.116.10:4083 , piece_id=GW5CGV5RHYMG6YQSFQDODMKSTQ5U66H6VDSI3EN7BRQE7AVT5KCA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731471173 (584h7m26.602554208s) [noise=true , address=145.79.227.45:4231 , piece_id=EFVDYZHLA4FUXTT7AXZKF6CR5TSF3CZZZCT2RD4MYQAEJXCD7CMQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731556797 (584h7m26.602639832s) [noise=true , address=145.79.227.39:4015 , piece_id=NUXCWFRRYGCL7BYQPIIWVDSDFNI3M67UTPFMJRRONZFX7DRJKSHQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731821480 (584h7m26.602904515s) [noise=true , address=145.79.227.26:4653 , piece_id=PMZF2DXCV7APV35LJ3HWJKZ4CIFLGHBE3H3FAIMIV26VA6XREWLA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731831490 (584h7m26.602914535s) [noise=true , address=145.79.227.2:4115 , piece_id=7TBP62LSPPZCMGDJKGWZB7VVVH2FKFWPILFPCVTKVF4ASMBQXMYQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732105271 (584h7m26.603188306s) [noise=true , address=145.79.227.42:4121 , piece_id=4SHKS46MVN7XD3IFQJY2JNQNDUMFSUXBWIKUWXRCHZ74DFQOZA3Q, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732161498 (584h7m26.603244533s) [noise=true , address=77.90.31.19:4841 , piece_id=GE2YWDIWDXQ2DCQSBSNWP4VH43522W5KPQFNZHO2TLZ6M2CG6JHA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732397818 (584h7m26.603480863s) [noise=true , address=145.79.227.50:4374 , piece_id=2DZNQVUN5WMHOE3WS2AXHV3TEXRZM6ONDURUDYPX7C3YF3EFZVUA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732545950 (584h7m26.603628985s) [noise=true , address=45.91.116.13:4153 , piece_id=KC3NVWYGKAN4QNIIXDGTZJHPDMBAGRYSYTXYIGL3PITCR6TDW23Q, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732671189 (584h7m26.603754224s) [noise=true , address=145.79.227.32:4834 , piece_id=5PNZTJOZRVVSGR6YIJD56YBQFZ4JYEVWRA6MF4R73HNFZ2WFBDVA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732866911 (584h7m26.603949946s) [noise=true , address=145.79.227.16:4911 , piece_id=PT33MS235LE4HHSJHV7W6U3VSVXQAQACZSRMNIRBMY7ZD7T57Y4A, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732936904 (584h7m26.604019939s) [noise=true , address=145.79.227.43:4157 , piece_id=6VCEZAES3MRECQ24EY2LI6DOHGNWAMCSQJWAKFGQDJQKQJUHETIQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.733256072 (584h7m26.604339108s) [noise=true , address=145.79.227.37:4567 , piece_id=FBGQLI3PPMBRICCWLQYDN6G7IWIATH3ZJA7ERDBOER3ZF25MIJRQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.734282146 (584h7m26.605365181s) [noise=true , address=45.91.116.12:4226 , piece_id=6ENOBE3LIHW3NDTSNSNUE72QCKWR5I7ZSLH4OVH7DZOXQGPVUOXA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:28.423611975 (584h7m28.29469501s) [upload=512056]: Next returned nil segment Dec 29 23:36:28.423629177 (584h7m28.294712222s) [upload=512056]: Waiting for error group managing segments... Dec 29 23:36:28.704542473 (584h7m28.575625508s) [noise=true , address=145.79.227.45:4231 , piece_id=EFVDYZHLA4FUXTT7AXZKF6CR5TSF3CZZZCT2RD4MYQAEJXCD7CMQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.719091847 (584h7m28.590174882s) [noise=true , address=145.79.227.46:4602 , piece_id=BSZRANNWJP2EZ2R74LXX4NSJXWHIEUTHMKIALZBW7ZFKUVXVQAKQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.722044971 (584h7m28.593128006s) [noise=true , address=145.79.227.16:4911 , piece_id=PT33MS235LE4HHSJHV7W6U3VSVXQAQACZSRMNIRBMY7ZD7T57Y4A, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.722945024 (584h7m28.594028059s) [noise=true , address=145.79.227.37:4567 , piece_id=FBGQLI3PPMBRICCWLQYDN6G7IWIATH3ZJA7ERDBOER3ZF25MIJRQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.725132810 (584h7m28.596215845s) [noise=true , address=145.79.227.42:4121 , piece_id=4SHKS46MVN7XD3IFQJY2JNQNDUMFSUXBWIKUWXRCHZ74DFQOZA3Q, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.729168825 (584h7m28.60025187s) [noise=true , address=145.79.227.39:4015 , piece_id=NUXCWFRRYGCL7BYQPIIWVDSDFNI3M67UTPFMJRRONZFX7DRJKSHQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.732318592 (584h7m28.603401627s) [noise=true , address=145.79.227.50:4374 , piece_id=2DZNQVUN5WMHOE3WS2AXHV3TEXRZM6ONDURUDYPX7C3YF3EFZVUA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.733290773 (584h7m28.604373808s) [noise=true , address=145.79.227.32:4834 , piece_id=5PNZTJOZRVVSGR6YIJD56YBQFZ4JYEVWRA6MF4R73HNFZ2WFBDVA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.736225501 (584h7m28.607308536s) [noise=true , address=145.79.227.2:4115 , piece_id=7TBP62LSPPZCMGDJKGWZB7VVVH2FKFWPILFPCVTKVF4ASMBQXMYQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.740314017 (584h7m28.611397062s) [noise=true , address=145.79.227.26:4653 , piece_id=PMZF2DXCV7APV35LJ3HWJKZ4CIFLGHBE3H3FAIMIV26VA6XREWLA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.902996515 (584h7m28.77407955s) [noise=true , address=77.90.31.19:4841 , piece_id=GE2YWDIWDXQ2DCQSBSNWP4VH43522W5KPQFNZHO2TLZ6M2CG6JHA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.917554686 (584h7m28.788637721s) [noise=true , address=77.90.31.14:4664 , piece_id=CRF4HS5WWUVREG3L4MN5ZDSIXUSXJP7OCLODRSL4MD2ME2BZUWVQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.946469964 (584h7m28.81755301s) [noise=true , address=45.91.116.10:4083 , piece_id=GW5CGV5RHYMG6YQSFQDODMKSTQ5U66H6VDSI3EN7BRQE7AVT5KCA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.946489963 (584h7m28.817572998s) [seg_pos={0 0}, upload=512056]: Segment reached optimal threshold of 13 pieces. Dec 29 23:36:28.946671118 (584h7m28.817754153s) [noise=true , address=45.91.116.12:4226 , piece_id=6ENOBE3LIHW3NDTSNSNUE72QCKWR5I7ZSLH4OVH7DZOXQGPVUOXA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:45454->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:45454->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45454->45.91.116.12:4226: use of closed network connection Dec 29 23:36:28.946678562 (584h7m28.817761597s) [noise=true , address=45.91.116.12:4226 , piece_id=6ENOBE3LIHW3NDTSNSNUE72QCKWR5I7ZSLH4OVH7DZOXQGPVUOXA, seg_pos={0 0}, upload=512056]: Long tail cancellation due to optimal threshold Dec 29 23:36:28.946830993 (584h7m28.817914028s) [noise=true , address=45.91.116.13:4153 , piece_id=KC3NVWYGKAN4QNIIXDGTZJHPDMBAGRYSYTXYIGL3PITCR6TDW23Q, seg_pos={0 0}, upload=512056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:56006->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:56006->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56006->45.91.116.13:4153: use of closed network connection Dec 29 23:36:28.946848996 (584h7m28.817932041s) [noise=true , address=45.91.116.13:4153 , piece_id=KC3NVWYGKAN4QNIIXDGTZJHPDMBAGRYSYTXYIGL3PITCR6TDW23Q, seg_pos={0 0}, upload=512056]: Long tail cancellation due to optimal threshold Dec 29 23:36:28.946877441 (584h7m28.817960486s) [noise=true , address=145.79.227.43:4157 , piece_id=6VCEZAES3MRECQ24EY2LI6DOHGNWAMCSQJWAKFGQDJQKQJUHETIQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 29 23:36:28.946884774 (584h7m28.817967809s) [noise=true , address=145.79.227.43:4157 , piece_id=6VCEZAES3MRECQ24EY2LI6DOHGNWAMCSQJWAKFGQDJQKQJUHETIQ, seg_pos={0 0}, upload=512056]: Long tail cancellation due to optimal threshold Dec 29 23:36:28.946962663 (584h7m28.818045699s) [seg_pos={0 0}, upload=512056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:36:28.947026865 (584h7m28.8181099s) [upload=512056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:36:28.947031034 (584h7m28.818114069s) [upload=512056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:36:29.231809329 (584h7m29.102892364s) [upload=512056]: Done uploading segments... Dec 29 23:36:29.231829437 (584h7m29.102912482s) [upload=512056]: Upload finished. err: <nil> Duration: 17.948499071s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........X..X..X....X.....X......XX...X...X..XXX...........] duration: 5.995070021s trace: https://jaeger.dev.storj.io/trace/2215ac2420cc0f18 FAIL (2025-12-29 23:44:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: GW5CGV5RHYMG6YQSFQDODMKSTQ5U66H6VDSI3EN7BRQE7AVT5KCA) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: CRF4HS5WWUVREG3L4MN5ZDSIXUSXJP7OCLODRSL4MD2ME2BZUWVQ) context deadline exceeded ======LOGS====== h7m26.358879078s) [upload=512056]: Uploading segments... Dec 29 23:36:26.487851669 (584h7m26.358934714s) [upload=512056]: Got next segment. Inline: false Dec 29 23:36:26.487880584 (584h7m26.358963629s) [upload=512056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:36:26.487884982 (584h7m26.358968017s) [upload=512056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:36:26.730337976 (584h7m26.601421011s) [seg_pos={0 0}, upload=512056]: Begin upload segment... Dec 29 23:36:26.730457603 (584h7m26.601540638s) [seg_pos={0 0}, upload=512056]: Done begin upload segment. Dec 29 23:36:26.730491598 (584h7m26.601574643s) [noise=true , address=77.90.31.14:4664 , piece_id=CRF4HS5WWUVREG3L4MN5ZDSIXUSXJP7OCLODRSL4MD2ME2BZUWVQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.730744679 (584h7m26.601827725s) [noise=true , address=145.79.227.46:4602 , piece_id=BSZRANNWJP2EZ2R74LXX4NSJXWHIEUTHMKIALZBW7ZFKUVXVQAKQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731249752 (584h7m26.602332797s) [noise=true , address=45.91.116.10:4083 , piece_id=GW5CGV5RHYMG6YQSFQDODMKSTQ5U66H6VDSI3EN7BRQE7AVT5KCA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731471173 (584h7m26.602554208s) [noise=true , address=145.79.227.45:4231 , piece_id=EFVDYZHLA4FUXTT7AXZKF6CR5TSF3CZZZCT2RD4MYQAEJXCD7CMQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731556797 (584h7m26.602639832s) [noise=true , address=145.79.227.39:4015 , piece_id=NUXCWFRRYGCL7BYQPIIWVDSDFNI3M67UTPFMJRRONZFX7DRJKSHQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731821480 (584h7m26.602904515s) [noise=true , address=145.79.227.26:4653 , piece_id=PMZF2DXCV7APV35LJ3HWJKZ4CIFLGHBE3H3FAIMIV26VA6XREWLA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731831490 (584h7m26.602914535s) [noise=true , address=145.79.227.2:4115 , piece_id=7TBP62LSPPZCMGDJKGWZB7VVVH2FKFWPILFPCVTKVF4ASMBQXMYQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732105271 (584h7m26.603188306s) [noise=true , address=145.79.227.42:4121 , piece_id=4SHKS46MVN7XD3IFQJY2JNQNDUMFSUXBWIKUWXRCHZ74DFQOZA3Q, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732161498 (584h7m26.603244533s) [noise=true , address=77.90.31.19:4841 , piece_id=GE2YWDIWDXQ2DCQSBSNWP4VH43522W5KPQFNZHO2TLZ6M2CG6JHA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732397818 (584h7m26.603480863s) [noise=true , address=145.79.227.50:4374 , piece_id=2DZNQVUN5WMHOE3WS2AXHV3TEXRZM6ONDURUDYPX7C3YF3EFZVUA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732545950 (584h7m26.603628985s) [noise=true , address=45.91.116.13:4153 , piece_id=KC3NVWYGKAN4QNIIXDGTZJHPDMBAGRYSYTXYIGL3PITCR6TDW23Q, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732671189 (584h7m26.603754224s) [noise=true , address=145.79.227.32:4834 , piece_id=5PNZTJOZRVVSGR6YIJD56YBQFZ4JYEVWRA6MF4R73HNFZ2WFBDVA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732866911 (584h7m26.603949946s) [noise=true , address=145.79.227.16:4911 , piece_id=PT33MS235LE4HHSJHV7W6U3VSVXQAQACZSRMNIRBMY7ZD7T57Y4A, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732936904 (584h7m26.604019939s) [noise=true , address=145.79.227.43:4157 , piece_id=6VCEZAES3MRECQ24EY2LI6DOHGNWAMCSQJWAKFGQDJQKQJUHETIQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.733256072 (584h7m26.604339108s) [noise=true , address=145.79.227.37:4567 , piece_id=FBGQLI3PPMBRICCWLQYDN6G7IWIATH3ZJA7ERDBOER3ZF25MIJRQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.734282146 (584h7m26.605365181s) [noise=true , address=45.91.116.12:4226 , piece_id=6ENOBE3LIHW3NDTSNSNUE72QCKWR5I7ZSLH4OVH7DZOXQGPVUOXA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:28.423611975 (584h7m28.29469501s) [upload=512056]: Next returned nil segment Dec 29 23:36:28.423629177 (584h7m28.294712222s) [upload=512056]: Waiting for error group managing segments... Dec 29 23:36:28.704542473 (584h7m28.575625508s) [noise=true , address=145.79.227.45:4231 , piece_id=EFVDYZHLA4FUXTT7AXZKF6CR5TSF3CZZZCT2RD4MYQAEJXCD7CMQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.719091847 (584h7m28.590174882s) [noise=true , address=145.79.227.46:4602 , piece_id=BSZRANNWJP2EZ2R74LXX4NSJXWHIEUTHMKIALZBW7ZFKUVXVQAKQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.722044971 (584h7m28.593128006s) [noise=true , address=145.79.227.16:4911 , piece_id=PT33MS235LE4HHSJHV7W6U3VSVXQAQACZSRMNIRBMY7ZD7T57Y4A, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.722945024 (584h7m28.594028059s) [noise=true , address=145.79.227.37:4567 , piece_id=FBGQLI3PPMBRICCWLQYDN6G7IWIATH3ZJA7ERDBOER3ZF25MIJRQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.725132810 (584h7m28.596215845s) [noise=true , address=145.79.227.42:4121 , piece_id=4SHKS46MVN7XD3IFQJY2JNQNDUMFSUXBWIKUWXRCHZ74DFQOZA3Q, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.729168825 (584h7m28.60025187s) [noise=true , address=145.79.227.39:4015 , piece_id=NUXCWFRRYGCL7BYQPIIWVDSDFNI3M67UTPFMJRRONZFX7DRJKSHQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.732318592 (584h7m28.603401627s) [noise=true , address=145.79.227.50:4374 , piece_id=2DZNQVUN5WMHOE3WS2AXHV3TEXRZM6ONDURUDYPX7C3YF3EFZVUA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.733290773 (584h7m28.604373808s) [noise=true , address=145.79.227.32:4834 , piece_id=5PNZTJOZRVVSGR6YIJD56YBQFZ4JYEVWRA6MF4R73HNFZ2WFBDVA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.736225501 (584h7m28.607308536s) [noise=true , address=145.79.227.2:4115 , piece_id=7TBP62LSPPZCMGDJKGWZB7VVVH2FKFWPILFPCVTKVF4ASMBQXMYQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.740314017 (584h7m28.611397062s) [noise=true , address=145.79.227.26:4653 , piece_id=PMZF2DXCV7APV35LJ3HWJKZ4CIFLGHBE3H3FAIMIV26VA6XREWLA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.902996515 (584h7m28.77407955s) [noise=true , address=77.90.31.19:4841 , piece_id=GE2YWDIWDXQ2DCQSBSNWP4VH43522W5KPQFNZHO2TLZ6M2CG6JHA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.917554686 (584h7m28.788637721s) [noise=true , address=77.90.31.14:4664 , piece_id=CRF4HS5WWUVREG3L4MN5ZDSIXUSXJP7OCLODRSL4MD2ME2BZUWVQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.946469964 (584h7m28.81755301s) [noise=true , address=45.91.116.10:4083 , piece_id=GW5CGV5RHYMG6YQSFQDODMKSTQ5U66H6VDSI3EN7BRQE7AVT5KCA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.946489963 (584h7m28.817572998s) [seg_pos={0 0}, upload=512056]: Segment reached optimal threshold of 13 pieces. Dec 29 23:36:28.946671118 (584h7m28.817754153s) [noise=true , address=45.91.116.12:4226 , piece_id=6ENOBE3LIHW3NDTSNSNUE72QCKWR5I7ZSLH4OVH7DZOXQGPVUOXA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:45454->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:45454->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45454->45.91.116.12:4226: use of closed network connection Dec 29 23:36:28.946678562 (584h7m28.817761597s) [noise=true , address=45.91.116.12:4226 , piece_id=6ENOBE3LIHW3NDTSNSNUE72QCKWR5I7ZSLH4OVH7DZOXQGPVUOXA, seg_pos={0 0}, upload=512056]: Long tail cancellation due to optimal threshold Dec 29 23:36:28.946830993 (584h7m28.817914028s) [noise=true , address=45.91.116.13:4153 , piece_id=KC3NVWYGKAN4QNIIXDGTZJHPDMBAGRYSYTXYIGL3PITCR6TDW23Q, seg_pos={0 0}, upload=512056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:56006->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:56006->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56006->45.91.116.13:4153: use of closed network connection Dec 29 23:36:28.946848996 (584h7m28.817932041s) [noise=true , address=45.91.116.13:4153 , piece_id=KC3NVWYGKAN4QNIIXDGTZJHPDMBAGRYSYTXYIGL3PITCR6TDW23Q, seg_pos={0 0}, upload=512056]: Long tail cancellation due to optimal threshold Dec 29 23:36:28.946877441 (584h7m28.817960486s) [noise=true , address=145.79.227.43:4157 , piece_id=6VCEZAES3MRECQ24EY2LI6DOHGNWAMCSQJWAKFGQDJQKQJUHETIQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 29 23:36:28.946884774 (584h7m28.817967809s) [noise=true , address=145.79.227.43:4157 , piece_id=6VCEZAES3MRECQ24EY2LI6DOHGNWAMCSQJWAKFGQDJQKQJUHETIQ, seg_pos={0 0}, upload=512056]: Long tail cancellation due to optimal threshold Dec 29 23:36:28.946962663 (584h7m28.818045699s) [seg_pos={0 0}, upload=512056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:36:28.947026865 (584h7m28.8181099s) [upload=512056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:36:28.947031034 (584h7m28.818114069s) [upload=512056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:36:29.231809329 (584h7m29.102892364s) [upload=512056]: Done uploading segments... Dec 29 23:36:29.231829437 (584h7m29.102912482s) [upload=512056]: Upload finished. err: <nil> Duration: 5.995098175s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...........X..X..X....X.....X......XX...X...X..XXX..........] duration: 6.078898271s trace: https://jaeger.dev.storj.io/trace/2215ac2420cc0f18 FAIL (2025-12-29 23:43:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: GW5CGV5RHYMG6YQSFQDODMKSTQ5U66H6VDSI3EN7BRQE7AVT5KCA) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: CRF4HS5WWUVREG3L4MN5ZDSIXUSXJP7OCLODRSL4MD2ME2BZUWVQ) context deadline exceeded ======LOGS====== h7m26.358879078s) [upload=512056]: Uploading segments... Dec 29 23:36:26.487851669 (584h7m26.358934714s) [upload=512056]: Got next segment. Inline: false Dec 29 23:36:26.487880584 (584h7m26.358963629s) [upload=512056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:36:26.487884982 (584h7m26.358968017s) [upload=512056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:36:26.730337976 (584h7m26.601421011s) [seg_pos={0 0}, upload=512056]: Begin upload segment... Dec 29 23:36:26.730457603 (584h7m26.601540638s) [seg_pos={0 0}, upload=512056]: Done begin upload segment. Dec 29 23:36:26.730491598 (584h7m26.601574643s) [noise=true , address=77.90.31.14:4664 , piece_id=CRF4HS5WWUVREG3L4MN5ZDSIXUSXJP7OCLODRSL4MD2ME2BZUWVQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.730744679 (584h7m26.601827725s) [noise=true , address=145.79.227.46:4602 , piece_id=BSZRANNWJP2EZ2R74LXX4NSJXWHIEUTHMKIALZBW7ZFKUVXVQAKQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731249752 (584h7m26.602332797s) [noise=true , address=45.91.116.10:4083 , piece_id=GW5CGV5RHYMG6YQSFQDODMKSTQ5U66H6VDSI3EN7BRQE7AVT5KCA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731471173 (584h7m26.602554208s) [noise=true , address=145.79.227.45:4231 , piece_id=EFVDYZHLA4FUXTT7AXZKF6CR5TSF3CZZZCT2RD4MYQAEJXCD7CMQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731556797 (584h7m26.602639832s) [noise=true , address=145.79.227.39:4015 , piece_id=NUXCWFRRYGCL7BYQPIIWVDSDFNI3M67UTPFMJRRONZFX7DRJKSHQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731821480 (584h7m26.602904515s) [noise=true , address=145.79.227.26:4653 , piece_id=PMZF2DXCV7APV35LJ3HWJKZ4CIFLGHBE3H3FAIMIV26VA6XREWLA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731831490 (584h7m26.602914535s) [noise=true , address=145.79.227.2:4115 , piece_id=7TBP62LSPPZCMGDJKGWZB7VVVH2FKFWPILFPCVTKVF4ASMBQXMYQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732105271 (584h7m26.603188306s) [noise=true , address=145.79.227.42:4121 , piece_id=4SHKS46MVN7XD3IFQJY2JNQNDUMFSUXBWIKUWXRCHZ74DFQOZA3Q, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732161498 (584h7m26.603244533s) [noise=true , address=77.90.31.19:4841 , piece_id=GE2YWDIWDXQ2DCQSBSNWP4VH43522W5KPQFNZHO2TLZ6M2CG6JHA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732397818 (584h7m26.603480863s) [noise=true , address=145.79.227.50:4374 , piece_id=2DZNQVUN5WMHOE3WS2AXHV3TEXRZM6ONDURUDYPX7C3YF3EFZVUA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732545950 (584h7m26.603628985s) [noise=true , address=45.91.116.13:4153 , piece_id=KC3NVWYGKAN4QNIIXDGTZJHPDMBAGRYSYTXYIGL3PITCR6TDW23Q, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732671189 (584h7m26.603754224s) [noise=true , address=145.79.227.32:4834 , piece_id=5PNZTJOZRVVSGR6YIJD56YBQFZ4JYEVWRA6MF4R73HNFZ2WFBDVA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732866911 (584h7m26.603949946s) [noise=true , address=145.79.227.16:4911 , piece_id=PT33MS235LE4HHSJHV7W6U3VSVXQAQACZSRMNIRBMY7ZD7T57Y4A, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732936904 (584h7m26.604019939s) [noise=true , address=145.79.227.43:4157 , piece_id=6VCEZAES3MRECQ24EY2LI6DOHGNWAMCSQJWAKFGQDJQKQJUHETIQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.733256072 (584h7m26.604339108s) [noise=true , address=145.79.227.37:4567 , piece_id=FBGQLI3PPMBRICCWLQYDN6G7IWIATH3ZJA7ERDBOER3ZF25MIJRQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.734282146 (584h7m26.605365181s) [noise=true , address=45.91.116.12:4226 , piece_id=6ENOBE3LIHW3NDTSNSNUE72QCKWR5I7ZSLH4OVH7DZOXQGPVUOXA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:28.423611975 (584h7m28.29469501s) [upload=512056]: Next returned nil segment Dec 29 23:36:28.423629177 (584h7m28.294712222s) [upload=512056]: Waiting for error group managing segments... Dec 29 23:36:28.704542473 (584h7m28.575625508s) [noise=true , address=145.79.227.45:4231 , piece_id=EFVDYZHLA4FUXTT7AXZKF6CR5TSF3CZZZCT2RD4MYQAEJXCD7CMQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.719091847 (584h7m28.590174882s) [noise=true , address=145.79.227.46:4602 , piece_id=BSZRANNWJP2EZ2R74LXX4NSJXWHIEUTHMKIALZBW7ZFKUVXVQAKQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.722044971 (584h7m28.593128006s) [noise=true , address=145.79.227.16:4911 , piece_id=PT33MS235LE4HHSJHV7W6U3VSVXQAQACZSRMNIRBMY7ZD7T57Y4A, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.722945024 (584h7m28.594028059s) [noise=true , address=145.79.227.37:4567 , piece_id=FBGQLI3PPMBRICCWLQYDN6G7IWIATH3ZJA7ERDBOER3ZF25MIJRQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.725132810 (584h7m28.596215845s) [noise=true , address=145.79.227.42:4121 , piece_id=4SHKS46MVN7XD3IFQJY2JNQNDUMFSUXBWIKUWXRCHZ74DFQOZA3Q, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.729168825 (584h7m28.60025187s) [noise=true , address=145.79.227.39:4015 , piece_id=NUXCWFRRYGCL7BYQPIIWVDSDFNI3M67UTPFMJRRONZFX7DRJKSHQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.732318592 (584h7m28.603401627s) [noise=true , address=145.79.227.50:4374 , piece_id=2DZNQVUN5WMHOE3WS2AXHV3TEXRZM6ONDURUDYPX7C3YF3EFZVUA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.733290773 (584h7m28.604373808s) [noise=true , address=145.79.227.32:4834 , piece_id=5PNZTJOZRVVSGR6YIJD56YBQFZ4JYEVWRA6MF4R73HNFZ2WFBDVA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.736225501 (584h7m28.607308536s) [noise=true , address=145.79.227.2:4115 , piece_id=7TBP62LSPPZCMGDJKGWZB7VVVH2FKFWPILFPCVTKVF4ASMBQXMYQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.740314017 (584h7m28.611397062s) [noise=true , address=145.79.227.26:4653 , piece_id=PMZF2DXCV7APV35LJ3HWJKZ4CIFLGHBE3H3FAIMIV26VA6XREWLA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.902996515 (584h7m28.77407955s) [noise=true , address=77.90.31.19:4841 , piece_id=GE2YWDIWDXQ2DCQSBSNWP4VH43522W5KPQFNZHO2TLZ6M2CG6JHA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.917554686 (584h7m28.788637721s) [noise=true , address=77.90.31.14:4664 , piece_id=CRF4HS5WWUVREG3L4MN5ZDSIXUSXJP7OCLODRSL4MD2ME2BZUWVQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.946469964 (584h7m28.81755301s) [noise=true , address=45.91.116.10:4083 , piece_id=GW5CGV5RHYMG6YQSFQDODMKSTQ5U66H6VDSI3EN7BRQE7AVT5KCA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.946489963 (584h7m28.817572998s) [seg_pos={0 0}, upload=512056]: Segment reached optimal threshold of 13 pieces. Dec 29 23:36:28.946671118 (584h7m28.817754153s) [noise=true , address=45.91.116.12:4226 , piece_id=6ENOBE3LIHW3NDTSNSNUE72QCKWR5I7ZSLH4OVH7DZOXQGPVUOXA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:45454->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:45454->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45454->45.91.116.12:4226: use of closed network connection Dec 29 23:36:28.946678562 (584h7m28.817761597s) [noise=true , address=45.91.116.12:4226 , piece_id=6ENOBE3LIHW3NDTSNSNUE72QCKWR5I7ZSLH4OVH7DZOXQGPVUOXA, seg_pos={0 0}, upload=512056]: Long tail cancellation due to optimal threshold Dec 29 23:36:28.946830993 (584h7m28.817914028s) [noise=true , address=45.91.116.13:4153 , piece_id=KC3NVWYGKAN4QNIIXDGTZJHPDMBAGRYSYTXYIGL3PITCR6TDW23Q, seg_pos={0 0}, upload=512056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:56006->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:56006->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56006->45.91.116.13:4153: use of closed network connection Dec 29 23:36:28.946848996 (584h7m28.817932041s) [noise=true , address=45.91.116.13:4153 , piece_id=KC3NVWYGKAN4QNIIXDGTZJHPDMBAGRYSYTXYIGL3PITCR6TDW23Q, seg_pos={0 0}, upload=512056]: Long tail cancellation due to optimal threshold Dec 29 23:36:28.946877441 (584h7m28.817960486s) [noise=true , address=145.79.227.43:4157 , piece_id=6VCEZAES3MRECQ24EY2LI6DOHGNWAMCSQJWAKFGQDJQKQJUHETIQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 29 23:36:28.946884774 (584h7m28.817967809s) [noise=true , address=145.79.227.43:4157 , piece_id=6VCEZAES3MRECQ24EY2LI6DOHGNWAMCSQJWAKFGQDJQKQJUHETIQ, seg_pos={0 0}, upload=512056]: Long tail cancellation due to optimal threshold Dec 29 23:36:28.946962663 (584h7m28.818045699s) [seg_pos={0 0}, upload=512056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:36:28.947026865 (584h7m28.8181099s) [upload=512056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:36:28.947031034 (584h7m28.818114069s) [upload=512056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:36:29.231809329 (584h7m29.102892364s) [upload=512056]: Done uploading segments... Dec 29 23:36:29.231829437 (584h7m29.102912482s) [upload=512056]: Upload finished. err: <nil> Duration: 6.07892356s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [............X..X..X....X.....X......XX...X...X..XXX.........] duration: 6.002528333s trace: https://jaeger.dev.storj.io/trace/2215ac2420cc0f18 FAIL (2025-12-29 23:43:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: GW5CGV5RHYMG6YQSFQDODMKSTQ5U66H6VDSI3EN7BRQE7AVT5KCA) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: CRF4HS5WWUVREG3L4MN5ZDSIXUSXJP7OCLODRSL4MD2ME2BZUWVQ) context deadline exceeded ======LOGS====== h7m26.358879078s) [upload=512056]: Uploading segments... Dec 29 23:36:26.487851669 (584h7m26.358934714s) [upload=512056]: Got next segment. Inline: false Dec 29 23:36:26.487880584 (584h7m26.358963629s) [upload=512056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 23:36:26.487884982 (584h7m26.358968017s) [upload=512056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 23:36:26.730337976 (584h7m26.601421011s) [seg_pos={0 0}, upload=512056]: Begin upload segment... Dec 29 23:36:26.730457603 (584h7m26.601540638s) [seg_pos={0 0}, upload=512056]: Done begin upload segment. Dec 29 23:36:26.730491598 (584h7m26.601574643s) [noise=true , address=77.90.31.14:4664 , piece_id=CRF4HS5WWUVREG3L4MN5ZDSIXUSXJP7OCLODRSL4MD2ME2BZUWVQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.730744679 (584h7m26.601827725s) [noise=true , address=145.79.227.46:4602 , piece_id=BSZRANNWJP2EZ2R74LXX4NSJXWHIEUTHMKIALZBW7ZFKUVXVQAKQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731249752 (584h7m26.602332797s) [noise=true , address=45.91.116.10:4083 , piece_id=GW5CGV5RHYMG6YQSFQDODMKSTQ5U66H6VDSI3EN7BRQE7AVT5KCA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731471173 (584h7m26.602554208s) [noise=true , address=145.79.227.45:4231 , piece_id=EFVDYZHLA4FUXTT7AXZKF6CR5TSF3CZZZCT2RD4MYQAEJXCD7CMQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731556797 (584h7m26.602639832s) [noise=true , address=145.79.227.39:4015 , piece_id=NUXCWFRRYGCL7BYQPIIWVDSDFNI3M67UTPFMJRRONZFX7DRJKSHQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731821480 (584h7m26.602904515s) [noise=true , address=145.79.227.26:4653 , piece_id=PMZF2DXCV7APV35LJ3HWJKZ4CIFLGHBE3H3FAIMIV26VA6XREWLA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.731831490 (584h7m26.602914535s) [noise=true , address=145.79.227.2:4115 , piece_id=7TBP62LSPPZCMGDJKGWZB7VVVH2FKFWPILFPCVTKVF4ASMBQXMYQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732105271 (584h7m26.603188306s) [noise=true , address=145.79.227.42:4121 , piece_id=4SHKS46MVN7XD3IFQJY2JNQNDUMFSUXBWIKUWXRCHZ74DFQOZA3Q, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732161498 (584h7m26.603244533s) [noise=true , address=77.90.31.19:4841 , piece_id=GE2YWDIWDXQ2DCQSBSNWP4VH43522W5KPQFNZHO2TLZ6M2CG6JHA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732397818 (584h7m26.603480863s) [noise=true , address=145.79.227.50:4374 , piece_id=2DZNQVUN5WMHOE3WS2AXHV3TEXRZM6ONDURUDYPX7C3YF3EFZVUA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732545950 (584h7m26.603628985s) [noise=true , address=45.91.116.13:4153 , piece_id=KC3NVWYGKAN4QNIIXDGTZJHPDMBAGRYSYTXYIGL3PITCR6TDW23Q, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732671189 (584h7m26.603754224s) [noise=true , address=145.79.227.32:4834 , piece_id=5PNZTJOZRVVSGR6YIJD56YBQFZ4JYEVWRA6MF4R73HNFZ2WFBDVA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732866911 (584h7m26.603949946s) [noise=true , address=145.79.227.16:4911 , piece_id=PT33MS235LE4HHSJHV7W6U3VSVXQAQACZSRMNIRBMY7ZD7T57Y4A, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.732936904 (584h7m26.604019939s) [noise=true , address=145.79.227.43:4157 , piece_id=6VCEZAES3MRECQ24EY2LI6DOHGNWAMCSQJWAKFGQDJQKQJUHETIQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.733256072 (584h7m26.604339108s) [noise=true , address=145.79.227.37:4567 , piece_id=FBGQLI3PPMBRICCWLQYDN6G7IWIATH3ZJA7ERDBOER3ZF25MIJRQ, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:26.734282146 (584h7m26.605365181s) [noise=true , address=45.91.116.12:4226 , piece_id=6ENOBE3LIHW3NDTSNSNUE72QCKWR5I7ZSLH4OVH7DZOXQGPVUOXA, seg_pos={0 0}, upload=512056]: Uploading piece... Dec 29 23:36:28.423611975 (584h7m28.29469501s) [upload=512056]: Next returned nil segment Dec 29 23:36:28.423629177 (584h7m28.294712222s) [upload=512056]: Waiting for error group managing segments... Dec 29 23:36:28.704542473 (584h7m28.575625508s) [noise=true , address=145.79.227.45:4231 , piece_id=EFVDYZHLA4FUXTT7AXZKF6CR5TSF3CZZZCT2RD4MYQAEJXCD7CMQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.719091847 (584h7m28.590174882s) [noise=true , address=145.79.227.46:4602 , piece_id=BSZRANNWJP2EZ2R74LXX4NSJXWHIEUTHMKIALZBW7ZFKUVXVQAKQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.722044971 (584h7m28.593128006s) [noise=true , address=145.79.227.16:4911 , piece_id=PT33MS235LE4HHSJHV7W6U3VSVXQAQACZSRMNIRBMY7ZD7T57Y4A, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.722945024 (584h7m28.594028059s) [noise=true , address=145.79.227.37:4567 , piece_id=FBGQLI3PPMBRICCWLQYDN6G7IWIATH3ZJA7ERDBOER3ZF25MIJRQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.725132810 (584h7m28.596215845s) [noise=true , address=145.79.227.42:4121 , piece_id=4SHKS46MVN7XD3IFQJY2JNQNDUMFSUXBWIKUWXRCHZ74DFQOZA3Q, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.729168825 (584h7m28.60025187s) [noise=true , address=145.79.227.39:4015 , piece_id=NUXCWFRRYGCL7BYQPIIWVDSDFNI3M67UTPFMJRRONZFX7DRJKSHQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.732318592 (584h7m28.603401627s) [noise=true , address=145.79.227.50:4374 , piece_id=2DZNQVUN5WMHOE3WS2AXHV3TEXRZM6ONDURUDYPX7C3YF3EFZVUA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.733290773 (584h7m28.604373808s) [noise=true , address=145.79.227.32:4834 , piece_id=5PNZTJOZRVVSGR6YIJD56YBQFZ4JYEVWRA6MF4R73HNFZ2WFBDVA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.736225501 (584h7m28.607308536s) [noise=true , address=145.79.227.2:4115 , piece_id=7TBP62LSPPZCMGDJKGWZB7VVVH2FKFWPILFPCVTKVF4ASMBQXMYQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.740314017 (584h7m28.611397062s) [noise=true , address=145.79.227.26:4653 , piece_id=PMZF2DXCV7APV35LJ3HWJKZ4CIFLGHBE3H3FAIMIV26VA6XREWLA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.902996515 (584h7m28.77407955s) [noise=true , address=77.90.31.19:4841 , piece_id=GE2YWDIWDXQ2DCQSBSNWP4VH43522W5KPQFNZHO2TLZ6M2CG6JHA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.917554686 (584h7m28.788637721s) [noise=true , address=77.90.31.14:4664 , piece_id=CRF4HS5WWUVREG3L4MN5ZDSIXUSXJP7OCLODRSL4MD2ME2BZUWVQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.946469964 (584h7m28.81755301s) [noise=true , address=45.91.116.10:4083 , piece_id=GW5CGV5RHYMG6YQSFQDODMKSTQ5U66H6VDSI3EN7BRQE7AVT5KCA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: <nil> Dec 29 23:36:28.946489963 (584h7m28.817572998s) [seg_pos={0 0}, upload=512056]: Segment reached optimal threshold of 13 pieces. Dec 29 23:36:28.946671118 (584h7m28.817754153s) [noise=true , address=45.91.116.12:4226 , piece_id=6ENOBE3LIHW3NDTSNSNUE72QCKWR5I7ZSLH4OVH7DZOXQGPVUOXA, seg_pos={0 0}, upload=512056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:45454->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:45454->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45454->45.91.116.12:4226: use of closed network connection Dec 29 23:36:28.946678562 (584h7m28.817761597s) [noise=true , address=45.91.116.12:4226 , piece_id=6ENOBE3LIHW3NDTSNSNUE72QCKWR5I7ZSLH4OVH7DZOXQGPVUOXA, seg_pos={0 0}, upload=512056]: Long tail cancellation due to optimal threshold Dec 29 23:36:28.946830993 (584h7m28.817914028s) [noise=true , address=45.91.116.13:4153 , piece_id=KC3NVWYGKAN4QNIIXDGTZJHPDMBAGRYSYTXYIGL3PITCR6TDW23Q, seg_pos={0 0}, upload=512056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:56006->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:56006->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56006->45.91.116.13:4153: use of closed network connection Dec 29 23:36:28.946848996 (584h7m28.817932041s) [noise=true , address=45.91.116.13:4153 , piece_id=KC3NVWYGKAN4QNIIXDGTZJHPDMBAGRYSYTXYIGL3PITCR6TDW23Q, seg_pos={0 0}, upload=512056]: Long tail cancellation due to optimal threshold Dec 29 23:36:28.946877441 (584h7m28.817960486s) [noise=true , address=145.79.227.43:4157 , piece_id=6VCEZAES3MRECQ24EY2LI6DOHGNWAMCSQJWAKFGQDJQKQJUHETIQ, seg_pos={0 0}, upload=512056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 29 23:36:28.946884774 (584h7m28.817967809s) [noise=true , address=145.79.227.43:4157 , piece_id=6VCEZAES3MRECQ24EY2LI6DOHGNWAMCSQJWAKFGQDJQKQJUHETIQ, seg_pos={0 0}, upload=512056]: Long tail cancellation due to optimal threshold Dec 29 23:36:28.946962663 (584h7m28.818045699s) [seg_pos={0 0}, upload=512056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 23:36:28.947026865 (584h7m28.8181099s) [upload=512056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 23:36:28.947031034 (584h7m28.818114069s) [upload=512056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 23:36:29.231809329 (584h7m29.102892364s) [upload=512056]: Done uploading segments... Dec 29 23:36:29.231829437 (584h7m29.102912482s) [upload=512056]: Upload finished. err: <nil> Duration: 6.002556786s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)