Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 869549 reports (Page 11600 of 17391)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X.....................X.............X...X.X....X.....] duration: 22.49992971s trace: https://jaeger.dev.storj.io/trace/29a389d9d0d6c11e FAIL (2025-12-09 14:09:07 UTC) orbiter warning: context deadline exceeded Duration: 22.499950211s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 4.681758985s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:09:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.681788944s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX..XXXXXXXX.XXXX..X.X.] duration: 4.347955747s trace: https://jaeger.dev.storj.io/trace/6b9abdd73d1cb197 FAIL (2025-12-09 14:08:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.347983128s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X....X............] duration: 10.91412662s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:08:56 UTC) orbiter warning: context deadline exceeded Duration: 10.914153701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 4.990429735s trace: https://jaeger.dev.storj.io/trace/7d316e68bfac4b21 FAIL (2025-12-09 14:08:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.990458729s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X....X...........] duration: 16.199658444s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:08:44 UTC) orbiter warning: context deadline exceeded Duration: 16.199727025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X..........X...X............X..] duration: 5.743980198s trace: https://jaeger.dev.storj.io/trace/435fea5a3b2801ab FAIL (2025-12-09 14:08:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.74400722s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [XXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX..XXXXXXXX.XXXX..X.X] duration: 5.000929688s trace: https://jaeger.dev.storj.io/trace/6b9abdd73d1cb197 FAIL (2025-12-09 14:08:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000961378s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [XXXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX..XXXXXXXX.XXXX..X.] duration: 3.931108233s trace: https://jaeger.dev.storj.io/trace/7cbb642003f517ad FAIL (2025-12-09 14:08:14 UTC) orbiter warning: context deadline exceeded Duration: 3.931142437s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 4.702618094s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:08:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.702648671s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.011200114s trace: https://jaeger.dev.storj.io/trace/7d316e68bfac4b21 FAIL (2025-12-09 14:08:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.01123544s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X.....................X.............X...X.X....X....] duration: 20.564953018s trace: https://jaeger.dev.storj.io/trace/29a389d9d0d6c11e FAIL (2025-12-09 14:07:53 UTC) orbiter warning: context deadline exceeded Duration: 20.564981052s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................X..........X...X............X.] duration: 5.677045491s trace: https://jaeger.dev.storj.io/trace/435fea5a3b2801ab FAIL (2025-12-09 14:07:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.677066011s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................X..........X...X............X] duration: 2m10.658668276s trace: https://jaeger.dev.storj.io/trace/435fea5a3b2801ab FAIL (2025-12-09 14:07:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2m10.658694547s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 4.792683106s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:07:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.792711899s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................................X....X..........] duration: 15.489537724s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:07:29 UTC) orbiter warning: context deadline exceeded Duration: 15.489569764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.162966132s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:07:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 5.162996138s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXXXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX..XXXXXXXX.XXXX..X] duration: 5.000871332s trace: https://jaeger.dev.storj.io/trace/7cbb642003f517ad FAIL (2025-12-09 14:07:17 UTC) orbiter warning: context deadline exceeded Duration: 5.00089681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..................................X........................] duration: 4.824900254s trace: https://jaeger.dev.storj.io/trace/7d316e68bfac4b21 FAIL (2025-12-09 14:06:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.824928969s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X.....................X.............X...X.X....X...] duration: 29.501972024s trace: https://jaeger.dev.storj.io/trace/29a389d9d0d6c11e FAIL (2025-12-09 14:06:51 UTC) orbiter warning: context deadline exceeded Duration: 29.502001018s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X....X.........] duration: 28.521589181s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:06:37 UTC) orbiter warning: context deadline exceeded Duration: 28.521614981s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXXXXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX..XXXXXXXX.XXXX..] duration: 5.001635262s trace: https://jaeger.dev.storj.io/trace/7989c87519cc9db3 FAIL (2025-12-09 14:06:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00166521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.186319639s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:06:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 5.186349136s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X.............................................X....X........] duration: 13.30074996s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:05:59 UTC) orbiter warning: context deadline exceeded Duration: 13.300777283s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..................................X.......................] duration: 4.928786933s trace: https://jaeger.dev.storj.io/trace/7d316e68bfac4b21 FAIL (2025-12-09 14:05:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.92881675s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X.....................X.............X...X.X....X..] duration: 6.409777961s trace: https://jaeger.dev.storj.io/trace/29a389d9d0d6c11e FAIL (2025-12-09 14:05:25 UTC) orbiter warning: context deadline exceeded Duration: 6.409804791s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [.XXXXXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX..XXXXXXXX.XXXX.] duration: 4.559007241s trace: https://jaeger.dev.storj.io/trace/7989c87519cc9db3 FAIL (2025-12-09 14:05:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.559034142s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..................................X......................] duration: 5.235341498s trace: https://jaeger.dev.storj.io/trace/7d316e68bfac4b21 FAIL (2025-12-09 14:05:12 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.23537387s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [X.XXXXXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX..XXXXXXXX.XXXX] duration: 5.000961811s trace: https://jaeger.dev.storj.io/trace/7989c87519cc9db3 FAIL (2025-12-09 14:05:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001003791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 4.685207484s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:04:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.685270695s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X.............................................X....X.......] duration: 12.38167687s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:04:57 UTC) orbiter warning: context deadline exceeded Duration: 12.381715414s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X..........X...X............] duration: 7.159038608s trace: https://jaeger.dev.storj.io/trace/1a34b9ace98af575 FAIL (2025-12-09 14:04:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.159075418s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..................................X.....................] duration: 4.99026953s trace: https://jaeger.dev.storj.io/trace/7d316e68bfac4b21 FAIL (2025-12-09 14:04:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.990293355s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X.....................X.............X...X.X....X.] duration: 21.233136468s trace: https://jaeger.dev.storj.io/trace/29a389d9d0d6c11e FAIL (2025-12-09 14:04:26 UTC) orbiter warning: context deadline exceeded Duration: 21.233168771s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [.X.XXXXXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX..XXXXXXXX.XXX] duration: 5.0010946s trace: https://jaeger.dev.storj.io/trace/5aa964e785ceb91a FAIL (2025-12-09 14:04:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001120909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X..........X...X...........] duration: 6.968425455s trace: https://jaeger.dev.storj.io/trace/1a34b9ace98af575 FAIL (2025-12-09 14:04:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.968546466s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..................................X....................] duration: 5.022855526s trace: https://jaeger.dev.storj.io/trace/7d316e68bfac4b21 FAIL (2025-12-09 14:04:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.022907656s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X.............................................X....X......] duration: 10.224796612s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:03:53 UTC) orbiter warning: context deadline exceeded Duration: 10.224825558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 4.772297617s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:03:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.77241993s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..................................X...................] duration: 5.014467557s trace: https://jaeger.dev.storj.io/trace/7d316e68bfac4b21 FAIL (2025-12-09 14:03:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.0144948s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X..........X...X..........] duration: 6.992228985s trace: https://jaeger.dev.storj.io/trace/1a34b9ace98af575 FAIL (2025-12-09 14:03:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.992271416s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X.....................X.............X...X.X....X] duration: 30.001272286s trace: https://jaeger.dev.storj.io/trace/29a389d9d0d6c11e FAIL (2025-12-09 14:03:27 UTC) orbiter warning: context deadline exceeded Duration: 30.001300298s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...X.............................................X....X.....] duration: 12.797666745s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:03:27 UTC) orbiter warning: context deadline exceeded Duration: 12.79769556s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [..X.XXXXXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX..XXXXXXXX.XX] duration: 5.000812097s trace: https://jaeger.dev.storj.io/trace/7d298de1bf3fb2c0 FAIL (2025-12-09 14:03:21 UTC) orbiter warning: context deadline exceeded Duration: 5.000868695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X..........X...X.........] duration: 8.630773274s trace: https://jaeger.dev.storj.io/trace/1a34b9ace98af575 FAIL (2025-12-09 14:03:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 8.630799343s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.....X..................................X..................] duration: 4.780422941s trace: https://jaeger.dev.storj.io/trace/7d316e68bfac4b21 FAIL (2025-12-09 14:03:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.780461455s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [...X.XXXXXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX..XXXXXXXX.X] duration: 5.000692198s trace: https://jaeger.dev.storj.io/trace/2bd6ef6864f85c86 FAIL (2025-12-09 14:02:56 UTC) orbiter warning: context deadline exceeded Duration: 5.000833829s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 4.754815819s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:02:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.754924575s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X.....................X.............X...X.X....] duration: 9.221274915s trace: https://jaeger.dev.storj.io/trace/23e38f537d9757b FAIL (2025-12-09 14:02:28 UTC) orbiter warning: context deadline exceeded Duration: 9.221307366s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X.............................................X....X....] duration: 23.359935617s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:02:28 UTC) orbiter warning: context deadline exceeded Duration: 23.359963109s core uplink us-select-1 us1-placement-20

1
...
11598
11599
11600
11601
11602
...
17391