Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [........X.XXXXXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX..XXXXX] duration:
5.001661988s trace:
https://jaeger.dev.storj.io/trace/424008baf0d25939
FAIL (2025-12-09 13:59:24 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001691425s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................................X..........X...X....] duration:
7.042705696s trace:
https://jaeger.dev.storj.io/trace/1a34b9ace98af575
FAIL (2025-12-09 13:59:12 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.042737357s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [X................X.....................X.............X...X.X] duration:
30.001120084s trace:
https://jaeger.dev.storj.io/trace/23e38f537d9757b
FAIL (2025-12-09 13:58:59 UTC)
orbiter warning: context deadline exceeded
Duration: 30.00114403s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
4.757145148s trace:
https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7
FAIL (2025-12-09 13:58:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment
Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments...
Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces.
Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF
Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold
Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold
Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments...
Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil>
Duration: 4.757194122s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [X........X.XXXXXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX..XXXX] duration:
5.000724533s trace:
https://jaeger.dev.storj.io/trace/7c303e02b91c40bb
FAIL (2025-12-09 13:58:42 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000748658s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X.....X..................................X...........] duration:
5.050729496s trace:
https://jaeger.dev.storj.io/trace/7d316e68bfac4b21
FAIL (2025-12-09 13:58:34 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.050760123s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................................X..........X...X...] duration:
7.006165551s trace:
https://jaeger.dev.storj.io/trace/1a34b9ace98af575
FAIL (2025-12-09 13:58:30 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.006194195s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X................X.....................X.............X...X.] duration:
23.148293896s trace:
https://jaeger.dev.storj.io/trace/632c2f35005100dd
FAIL (2025-12-09 13:58:27 UTC)
orbiter warning: context deadline exceeded
Duration: 23.14832311s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [XX........X.XXXXXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX..XXX] duration:
5.001613466s trace:
https://jaeger.dev.storj.io/trace/689ed38a7b769563
FAIL (2025-12-09 13:58:10 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001635759s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...X.....X.............................................X....] duration:
16.977843607s trace:
https://jaeger.dev.storj.io/trace/3fa217b031b80ed0
FAIL (2025-12-09 13:58:02 UTC)
orbiter warning: context deadline exceeded
Duration: 16.977875477s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........................................X..........X...X..] duration:
7.215875821s trace:
https://jaeger.dev.storj.io/trace/1a34b9ace98af575
FAIL (2025-12-09 13:57:44 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.215891852s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X.....X..................................X..........] duration:
4.9900622s trace:
https://jaeger.dev.storj.io/trace/7d316e68bfac4b21
FAIL (2025-12-09 13:57:41 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.990092467s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
4.871172501s trace:
https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7
FAIL (2025-12-09 13:57:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment
Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments...
Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces.
Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF
Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold
Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold
Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments...
Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil>
Duration: 4.871199562s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X................X.....................X.............X...X] duration:
30.001254982s trace:
https://jaeger.dev.storj.io/trace/632c2f35005100dd
FAIL (2025-12-09 13:57:36 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001268508s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........X.....X..................................X.........] duration:
5.166145668s trace:
https://jaeger.dev.storj.io/trace/7d316e68bfac4b21
FAIL (2025-12-09 13:57:18 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.166178271s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....X.....X.............................................X...] duration:
20.616940466s trace:
https://jaeger.dev.storj.io/trace/3fa217b031b80ed0
FAIL (2025-12-09 13:57:14 UTC)
orbiter warning: context deadline exceeded
Duration: 20.616971926s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
32 out of 60 is failed (threshold=6) [.XX........X.XXXXXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX..XX] duration:
5.002441264s trace:
https://jaeger.dev.storj.io/trace/6b491fb977dade74
FAIL (2025-12-09 13:57:06 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002464749s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X................X.....................X.............X...] duration:
30.00105216s trace:
https://jaeger.dev.storj.io/trace/5e8f0d77ee22d09f
FAIL (2025-12-09 13:57:05 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001078329s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
5.128567824s trace:
https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7
FAIL (2025-12-09 13:56:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment
Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments...
Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces.
Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF
Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold
Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold
Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments...
Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil>
Duration: 5.12859169s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....X.....X.............................................X..] duration:
14.336569346s trace:
https://jaeger.dev.storj.io/trace/3fa217b031b80ed0
FAIL (2025-12-09 13:56:52 UTC)
orbiter warning: context deadline exceeded
Duration: 14.336605384s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........................................X..........X...X.] duration:
7.070737979s trace:
https://jaeger.dev.storj.io/trace/1a34b9ace98af575
FAIL (2025-12-09 13:56:35 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.070770371s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........X.....X..................................X........] duration:
4.99143804s trace:
https://jaeger.dev.storj.io/trace/7d316e68bfac4b21
FAIL (2025-12-09 13:56:33 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.991468728s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
12.882375059s trace:
https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7
FAIL (2025-12-09 13:56:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment
Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments...
Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces.
Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF
Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold
Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold
Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments...
Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil>
Duration: 12.882405686s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
32 out of 60 is failed (threshold=6) [X.XX........X.XXXXXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX..X] duration:
5.001424286s trace:
https://jaeger.dev.storj.io/trace/569cbb252162d2a2
FAIL (2025-12-09 13:56:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001448813s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......X.....X.............................................X.] duration:
11.492416057s trace:
https://jaeger.dev.storj.io/trace/3fa217b031b80ed0
FAIL (2025-12-09 13:56:07 UTC)
orbiter warning: context deadline exceeded
Duration: 11.492432068s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............................................X..........X...X] duration:
30.251307307s trace:
https://jaeger.dev.storj.io/trace/1a34b9ace98af575
FAIL (2025-12-09 13:55:56 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.251336202s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.......X.....X.............................................X] duration:
1m0.001580597s trace:
https://jaeger.dev.storj.io/trace/3fa217b031b80ed0
FAIL (2025-12-09 13:55:55 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001616535s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........X.....X..................................X.......] duration:
5.073442084s trace:
https://jaeger.dev.storj.io/trace/7d316e68bfac4b21
FAIL (2025-12-09 13:55:54 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.073467493s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
23.18242644s trace:
https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7
FAIL (2025-12-09 13:55:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece...
Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment
Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments...
Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil>
Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces.
Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF
Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold
Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold
Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments...
Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil>
Duration: 23.182475704s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X................X.....................X.............X..] duration:
13.727628214s trace:
https://jaeger.dev.storj.io/trace/5e8f0d77ee22d09f
FAIL (2025-12-09 13:55:35 UTC)
orbiter warning: context deadline exceeded
Duration: 13.727659994s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X..........X...] duration:
6.506695374s trace:
https://jaeger.dev.storj.io/trace/3762a1dcdc93a3f8
FAIL (2025-12-09 13:55:16 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.506723527s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
31 out of 60 is failed (threshold=6) [.X.XX........X.XXXXXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX..] duration:
4.166456253s trace:
https://jaeger.dev.storj.io/trace/517e0b3b3c3f7cd4
FAIL (2025-12-09 13:55:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.166487051s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............X.....X..................................X......] duration:
5.668181897s trace:
https://jaeger.dev.storj.io/trace/7d316e68bfac4b21
FAIL (2025-12-09 13:55:03 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.6682149s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X................X.....................X.............X.] duration:
14.6665655s trace:
https://jaeger.dev.storj.io/trace/5e8f0d77ee22d09f
FAIL (2025-12-09 13:54:57 UTC)
orbiter warning: context deadline exceeded
Duration: 14.666590368s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X..........X..] duration:
6.92463636s trace:
https://jaeger.dev.storj.io/trace/3762a1dcdc93a3f8
FAIL (2025-12-09 13:54:53 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.924663734s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............X.....X..................................X.....] duration:
5.851108588s trace:
https://jaeger.dev.storj.io/trace/7d316e68bfac4b21
FAIL (2025-12-09 13:54:42 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.851143334s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [......X................X.....................X.............X] duration:
30.001161766s trace:
https://jaeger.dev.storj.io/trace/5e8f0d77ee22d09f
FAIL (2025-12-09 13:54:42 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001193927s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
31 out of 60 is failed (threshold=6) [..X.XX........X.XXXXXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX.] duration:
4.729005146s trace:
https://jaeger.dev.storj.io/trace/517e0b3b3c3f7cd4
FAIL (2025-12-09 13:54:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.729029321s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............X.....X..................................X....] duration:
5.971279166s trace:
https://jaeger.dev.storj.io/trace/7d316e68bfac4b21
FAIL (2025-12-09 13:54:22 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.971319393s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........X.....X.............................................] duration:
6.597882488s trace:
https://jaeger.dev.storj.io/trace/60a7cbb2e1d6c052
FAIL (2025-12-09 13:54:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.597912325s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X................X.....................X.............] duration:
22.63667694s trace:
https://jaeger.dev.storj.io/trace/73f5ad26d23b6706
FAIL (2025-12-09 13:54:08 UTC)
orbiter warning: context deadline exceeded
Duration: 22.636694875s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.869359382s trace:
https://jaeger.dev.storj.io/trace/314592c187a50ec
FAIL (2025-12-09 13:53:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, Piece ID: MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ) context deadline exceeded; piecestore: (Node ID: 1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, Piece ID: G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ) context deadline exceeded
======LOGS======
Dec 9 13:08:34.623724641 (93h39m34.494807697s) [upload=81665]: Starting upload
Dec 9 13:08:34.623815704 (93h39m34.494898749s) [upload=81665]: Done starting upload
Dec 9 13:08:34.623966372 (93h39m34.495049417s) [upload=81665]: Uploading segments...
Dec 9 13:08:34.624323702 (93h39m34.495406747s) [upload=81665]: Got next segment. Inline: false
Dec 9 13:08:34.624371984 (93h39m34.495455029s) [upload=81665]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 9 13:08:34.624377203 (93h39m34.495460238s) [upload=81665]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 9 13:08:37.281127755 (93h39m37.15221081s) [seg_pos={0 0}, upload=81665]: Begin upload segment...
Dec 9 13:08:37.281189272 (93h39m37.152272317s) [seg_pos={0 0}, upload=81665]: Done begin upload segment.
Dec 9 13:08:37.281217316 (93h39m37.152300351s) [noise=true , address=145.79.227.43:4159 , piece_id=FEA675SNDNVPTWQ4AUDTQIPWUBCCIG2XUZMW5EB5KL2G4VPYEOUA, seg_pos={0 0}, upload=81665]: Uploading piece...
Dec 9 13:08:37.281227836 (93h39m37.152310881s) [noise=true , address=77.90.31.14:4663 , piece_id=R4HWRMWHTFSO4T4EOLGYF3UQDF4EWYL6I5WHRCVSGGAHSBSRJORA, seg_pos={0 0}, upload=81665]: Uploading piece...
Dec 9 13:08:37.282226888 (93h39m37.153309923s) [noise=true , address=145.79.227.40:4049 , piece_id=G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ, seg_pos={0 0}, upload=81665]: Uploading piece...
Dec 9 13:08:37.282666044 (93h39m37.153749079s) [noise=true , address=145.79.227.39:4015 , piece_id=YFUWDWIYLSGHPBIDRKXT7HLLN2VMXNQBL6DH7YFP7W4NXUP5THXA, seg_pos={0 0}, upload=81665]: Uploading piece...
Dec 9 13:08:37.283051929 (93h39m37.154134964s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Uploading piece...
Dec 9 13:08:37.283463072 (93h39m37.154546117s) [noise=true , address=145.79.227.27:4691 , piece_id=T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ, seg_pos={0 0}, upload=81665]: Uploading piece...
Dec 9 13:08:37.283879755 (93h39m37.1549628s) [noise=true , address=145.79.227.41:4085 , piece_id=V57CTYDJ7K7KRXJSGX5N4YZZND7TLAO5VS2K7FWSHG4X4LW2MB3Q, seg_pos={0 0}, upload=81665]: Uploading piece...
Dec 9 13:08:37.284332708 (93h39m37.155415753s) [noise=true , address=145.79.227.21:4474 , piece_id=JCR2AIDA5F4QSAWZ2QZF6L4BENU3Q6RVVEQ7MSLV2Z47Z4RCTVFA, seg_pos={0 0}, upload=81665]: Uploading piece...
Dec 9 13:08:37.285609359 (93h39m37.156692404s) [noise=true , address=145.79.227.26:4654 , piece_id=MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ, seg_pos={0 0}, upload=81665]: Uploading piece...
Dec 9 13:08:37.285915071 (93h39m37.156998116s) [noise=true , address=145.79.227.42:4123 , piece_id=J3YM7ROA4R6RTIFWPEW3EY25SUH6MM655ZNPL4V4BNJG2IXKVDRA, seg_pos={0 0}, upload=81665]: Uploading piece...
Dec 9 13:08:37.286766462 (93h39m37.157849497s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Uploading piece...
Dec 9 13:08:37.287035495 (93h39m37.15811853s) [noise=true , address=145.79.227.32:4836 , piece_id=RY7B33YGAHG6YL3TSQHKKLONKTAEUZKHXENM57QSDBMOPJDOTNVQ, seg_pos={0 0}, upload=81665]: Uploading piece...
Dec 9 13:08:37.287925931 (93h39m37.159008976s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Uploading piece...
Dec 9 13:08:37.288456982 (93h39m37.159540027s) [noise=true , address=145.79.227.31:4799 , piece_id=IY4OCZ5XUNH7U5VDLDMNBG4V5HMK5IZ7DLA2QNZ5SPJXSK3Y6GNA, seg_pos={0 0}, upload=81665]: Uploading piece...
Dec 9 13:08:37.288723048 (93h39m37.159806093s) [noise=true , address=145.79.227.34:4906 , piece_id=PPXK6KPYBWYCP52G3PTCWB2WIDUKR2LS4W3DPUTSY2KJLMTI64NQ, seg_pos={0 0}, upload=81665]: Uploading piece...
Dec 9 13:08:37.290630710 (93h39m37.161713775s) [noise=true , address=145.79.227.14:4838 , piece_id=M6F7XBYTGU2LU56ILGNMTHAA6ZFJKHF353CQEJB3HOHW6AG2YUEA, seg_pos={0 0}, upload=81665]: Uploading piece...
Dec 9 13:08:39.186350861 (93h39m39.057433906s) [upload=81665]: Next returned nil segment
Dec 9 13:08:39.186369836 (93h39m39.057452871s) [upload=81665]: Waiting for error group managing segments...
Dec 9 13:08:39.492788692 (93h39m39.363871737s) [noise=true , address=77.90.31.14:4663 , piece_id=R4HWRMWHTFSO4T4EOLGYF3UQDF4EWYL6I5WHRCVSGGAHSBSRJORA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil>
Dec 9 13:08:40.380040727 (93h39m40.251123782s) [noise=true , address=145.79.227.41:4085 , piece_id=V57CTYDJ7K7KRXJSGX5N4YZZND7TLAO5VS2K7FWSHG4X4LW2MB3Q, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil>
Dec 9 13:08:40.382027781 (93h39m40.253110826s) [noise=true , address=145.79.227.32:4836 , piece_id=RY7B33YGAHG6YL3TSQHKKLONKTAEUZKHXENM57QSDBMOPJDOTNVQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil>
Dec 9 13:08:40.388157955 (93h39m40.259241s) [noise=true , address=145.79.227.43:4159 , piece_id=FEA675SNDNVPTWQ4AUDTQIPWUBCCIG2XUZMW5EB5KL2G4VPYEOUA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil>
Dec 9 13:08:40.403198054 (93h39m40.274281099s) [noise=true , address=145.79.227.39:4015 , piece_id=YFUWDWIYLSGHPBIDRKXT7HLLN2VMXNQBL6DH7YFP7W4NXUP5THXA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil>
Dec 9 13:08:40.408072455 (93h39m40.27915549s) [noise=true , address=145.79.227.14:4838 , piece_id=M6F7XBYTGU2LU56ILGNMTHAA6ZFJKHF353CQEJB3HOHW6AG2YUEA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil>
Dec 9 13:08:40.416722717 (93h39m40.287805752s) [noise=true , address=145.79.227.31:4799 , piece_id=IY4OCZ5XUNH7U5VDLDMNBG4V5HMK5IZ7DLA2QNZ5SPJXSK3Y6GNA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil>
Dec 9 13:08:40.419255590 (93h39m40.290338625s) [noise=true , address=145.79.227.34:4906 , piece_id=PPXK6KPYBWYCP52G3PTCWB2WIDUKR2LS4W3DPUTSY2KJLMTI64NQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil>
Dec 9 13:08:40.434848462 (93h39m40.305931507s) [noise=true , address=145.79.227.21:4474 , piece_id=JCR2AIDA5F4QSAWZ2QZF6L4BENU3Q6RVVEQ7MSLV2Z47Z4RCTVFA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil>
Dec 9 13:08:40.463333804 (93h39m40.334416839s) [noise=true , address=145.79.227.27:4691 , piece_id=T4DHGDJHC4E3UPBCDINWBBYOLKE5Q4P6I577BI742RUO3IFOKDXQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil>
Dec 9 13:08:40.467513042 (93h39m40.338596087s) [noise=true , address=145.79.227.42:4123 , piece_id=J3YM7ROA4R6RTIFWPEW3EY25SUH6MM655ZNPL4V4BNJG2IXKVDRA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil>
Dec 9 13:08:40.471574776 (93h39m40.342657822s) [noise=true , address=145.79.227.26:4654 , piece_id=MA5AUA7S2ZQ33WVBUQMB5L6Q23MCIUUQLJBARUJSJTEYLBZMOISQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil>
Dec 9 13:08:40.586586735 (93h39m40.45766978s) [noise=true , address=145.79.227.40:4049 , piece_id=G3X4V53OIX2KJYBHYW5YVTA4LASTYOOIEKPMX3HH6IOYXSRM7XGQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: <nil>
Dec 9 13:08:40.586615669 (93h39m40.457698714s) [seg_pos={0 0}, upload=81665]: Segment reached optimal threshold of 13 pieces.
Dec 9 13:08:40.587192418 (93h39m40.458275463s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: read tcp 188.166.225.170:41030->145.79.227.30:4762: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 13:08:40.587212296 (93h39m40.458295341s) [noise=true , address=145.79.227.30:4762 , piece_id=SPBFNFHMT7V27ITQNMNEODB3F7N7QWQ6SVHQU4JJDZD5EA6IGEXA, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold
Dec 9 13:08:40.587265738 (93h39m40.458348793s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:33846->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 13:08:40.587278271 (93h39m40.458361316s) [noise=true , address=145.79.227.38:4977 , piece_id=P5XEHLFEVFE3PDVBFP4OE7HURJMVR62PTURD6VOLUTL5S66JT4OA, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold
Dec 9 13:08:40.587358154 (93h39m40.458441199s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: read tcp 188.166.225.170:35084->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 13:08:40.587366700 (93h39m40.458449745s) [noise=true , address=145.79.227.26:4655 , piece_id=MAPJZ3AFTCJ5E3PPWA4ZOOL4YAGM542QSSLL7W7BMNSL7SHSX5NQ, seg_pos={0 0}, upload=81665]: Long tail cancellation due to optimal threshold
Dec 9 13:08:40.587391287 (93h39m40.458474322s) [seg_pos={0 0}, upload=81665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 9 13:08:40.587514591 (93h39m40.458597626s) [upload=81665]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 13:08:40.587521073 (93h39m40.458604119s) [upload=81665]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 13:08:55.391439615 (93h39m55.26252267s) [upload=81665]: Done uploading segments...
Dec 9 13:08:55.391462578 (93h39m55.262545613s) [upload=81665]: Upload finished. err: <nil>
Duration: 5.869382375s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X..........X.] duration:
6.580013611s trace:
https://jaeger.dev.storj.io/trace/3762a1dcdc93a3f8
FAIL (2025-12-09 13:53:54 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.580032718s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................X..........X] duration:
35.875263749s trace:
https://jaeger.dev.storj.io/trace/3762a1dcdc93a3f8
FAIL (2025-12-09 13:53:47 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 35.875307332s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X................X.....................X............] duration:
26.30612587s trace:
https://jaeger.dev.storj.io/trace/73f5ad26d23b6706
FAIL (2025-12-09 13:53:45 UTC)
orbiter warning: context deadline exceeded
Duration: 26.306146188s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........X.....X............................................] duration:
12.921896796s trace:
https://jaeger.dev.storj.io/trace/60a7cbb2e1d6c052
FAIL (2025-12-09 13:53:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.921925402s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
31 out of 60 is failed (threshold=6) [...X.XX........X.XXXXXXXXX.XXXX.X.X....X.X..XXX.....XX.XXXXX] duration:
5.001014657s trace:
https://jaeger.dev.storj.io/trace/517e0b3b3c3f7cd4
FAIL (2025-12-09 13:53:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001039404s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
8.582073291s trace:
https://jaeger.dev.storj.io/trace/9e51647c84d1fb5
FAIL (2025-12-09 13:53:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.469453596 (93h37m4.340536641s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.469560099 (93h37m4.340643144s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.470052186 (93h37m4.341135231s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.470103654 (93h37m4.341186699s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.470503986 (93h37m4.341587031s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.470603416 (93h37m4.341686471s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.471108767 (93h37m4.342191832s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.471161488 (93h37m4.342244533s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.471570507 (93h37m4.342653562s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.471658214 (93h37m4.342741259s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.472164348 (93h37m4.343247393s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.472240212 (93h37m4.343323257s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.472826509 (93h37m4.343909554s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.472993457 (93h37m4.344076502s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:06.014093105 (93h37m5.88517615s) [upload=81626]: Next returned nil segment
Dec 9 13:06:06.014111329 (93h37m5.885194374s) [upload=81626]: Waiting for error group managing segments...
Dec 9 13:06:06.452227624 (93h37m6.323310659s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.477649473 (93h37m6.348732518s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.484904859 (93h37m6.355987894s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.488510093 (93h37m6.359593138s) [noise=true , address=77.90.31.18:4820 , piece_id=SUANM4P45VAKRA7RZRCXOKAU5MP5PY2CCSUEIJ3ECV4Y5NCRXCRQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.491439753 (93h37m6.362522798s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.498682043 (93h37m6.369765078s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.508793167 (93h37m6.379876202s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.527211489 (93h37m6.398294535s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.545142274 (93h37m6.416225319s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.585701106 (93h37m6.456784151s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:07.181237322 (93h37m7.052320368s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:07.249731199 (93h37m7.120814244s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:07.447190704 (93h37m7.318273749s) [noise=true , address=145.79.227.31:4809 , piece_id=NGGDCXTIWQ4UWG2XYZ76PW2CGWONS63WOAMAE5PU5BZVUWYU6HTQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:13.905270685 (93h37m13.77635372s) [noise=true , address=45.91.116.8:4071 , piece_id=4PVVWGWIDNVHIYHZNNSXECZ6VO4VHGQRSPJGOSAEEJ3WXNYQF6NQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:13.905292547 (93h37m13.776375592s) [seg_pos={0 0}, upload=81626]: Segment reached optimal threshold of 14 pieces.
Dec 9 13:06:13.905521141 (93h37m13.776604186s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E): protocol: expected piece hash; context canceled; EOF
Dec 9 13:06:13.905530960 (93h37m13.776613995s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold
Dec 9 13:06:13.905599190 (93h37m13.776682225s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection
Dec 9 13:06:13.905618116 (93h37m13.776701161s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold
Dec 9 13:06:13.905716824 (93h37m13.776799869s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection
Dec 9 13:06:13.905727003 (93h37m13.776810038s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold
Dec 9 13:06:13.905954988 (93h37m13.777038043s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W3qDq8BssS8V6osW9NwFHZHMJcCRfjYP8qKwphE5SenVewWXy): protocol: expected piece hash; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; EOF
Dec 9 13:06:13.905964836 (93h37m13.777047882s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold
Dec 9 13:06:13.906061531 (93h37m13.777144576s) [seg_pos={0 0}, upload=81626]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 9 13:06:13.906185627 (93h37m13.777268672s) [upload=81626]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 13:06:13.906191748 (93h37m13.777274793s) [upload=81626]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 13:07:02.540166267 (93h38m2.411249312s) [upload=81626]: Done uploading segments...
Dec 9 13:07:02.540284373 (93h38m2.411367428s) [upload=81626]: Upload finished. err: metaclient: context deadline exceeded
Duration: 8.582103347s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............X.....X..................................X...] duration:
5.668266151s trace:
https://jaeger.dev.storj.io/trace/7d316e68bfac4b21
FAIL (2025-12-09 13:53:08 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.668299605s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
10.936129841s trace:
https://jaeger.dev.storj.io/trace/9e51647c84d1fb5
FAIL (2025-12-09 13:52:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.469453596 (93h37m4.340536641s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.469560099 (93h37m4.340643144s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.470052186 (93h37m4.341135231s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.470103654 (93h37m4.341186699s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.470503986 (93h37m4.341587031s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.470603416 (93h37m4.341686471s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.471108767 (93h37m4.342191832s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.471161488 (93h37m4.342244533s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.471570507 (93h37m4.342653562s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.471658214 (93h37m4.342741259s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.472164348 (93h37m4.343247393s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.472240212 (93h37m4.343323257s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.472826509 (93h37m4.343909554s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:04.472993457 (93h37m4.344076502s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Uploading piece...
Dec 9 13:06:06.014093105 (93h37m5.88517615s) [upload=81626]: Next returned nil segment
Dec 9 13:06:06.014111329 (93h37m5.885194374s) [upload=81626]: Waiting for error group managing segments...
Dec 9 13:06:06.452227624 (93h37m6.323310659s) [noise=true , address=5.231.73.3:4013 , piece_id=K7OAMZRL77AB7B6SMS7QKQPCHFPG7WJ4VT6PKCYKGXESIJZFMI4A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.477649473 (93h37m6.348732518s) [noise=true , address=77.90.31.11:4560 , piece_id=SQT5FHVZ35OO6SJ2SSVGYUPTQRLIMVLP4EL3QLO24BYVFAJCCFPQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.484904859 (93h37m6.355987894s) [noise=true , address=77.90.31.10:4541 , piece_id=GZAZIFXVCXMAA7GW7O6ZUOM2UZ7GE5BJLV4W6VFIVTNF6LKQNPAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.488510093 (93h37m6.359593138s) [noise=true , address=77.90.31.18:4820 , piece_id=SUANM4P45VAKRA7RZRCXOKAU5MP5PY2CCSUEIJ3ECV4Y5NCRXCRQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.491439753 (93h37m6.362522798s) [noise=true , address=77.90.31.15:4712 , piece_id=5NC6YRYQYXKM37DUJ32O2ZUHMPI67M7CZJHTINFFI7FOS2UA3EKA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.498682043 (93h37m6.369765078s) [noise=true , address=77.90.31.14:4690 , piece_id=OQPKPJGWCHZ4VECNF5NZ5D5BUUVXQUSYU5V42URMVTAU7XMFJG2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.508793167 (93h37m6.379876202s) [noise=true , address=77.90.31.21:4892 , piece_id=E6MCLHXXEZJ5B4SQNREJ3MQGWDBNN4XHOPVJA4XTPEHSUEQU2I6Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.527211489 (93h37m6.398294535s) [noise=true , address=77.90.31.19:4855 , piece_id=PC76YIQ2T2VMKCMOCO44XRVWEOSXIYQSW626FUHLADLPTFJXOKAA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.545142274 (93h37m6.416225319s) [noise=true , address=77.90.31.16:4747 , piece_id=F2CH63LIUNGXIHHKVFO77RMBXVFBQR25Z2W3QFGAAEH5OOYB5R3A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:06.585701106 (93h37m6.456784151s) [noise=true , address=77.90.31.17:4800 , piece_id=KCLPHS2RKPVHG7E5Q5DVFAR6VXBIMPTVGPAULWA36L4OZSNYPAVA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:07.181237322 (93h37m7.052320368s) [noise=true , address=77.90.31.13:4640 , piece_id=BRSVAGQ3TBKGRSYSJ6XY6VRH2Z37F22IX2HBEZ7LVAMVF2IBYXXQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:07.249731199 (93h37m7.120814244s) [noise=true , address=77.90.31.12:4595 , piece_id=HJTGJCMDA42CVSLLYTLK4KS233K6PKK6EXH2KXPSMMKVBEIPFU2A, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:07.447190704 (93h37m7.318273749s) [noise=true , address=145.79.227.31:4809 , piece_id=NGGDCXTIWQ4UWG2XYZ76PW2CGWONS63WOAMAE5PU5BZVUWYU6HTQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:13.905270685 (93h37m13.77635372s) [noise=true , address=45.91.116.8:4071 , piece_id=4PVVWGWIDNVHIYHZNNSXECZ6VO4VHGQRSPJGOSAEEJ3WXNYQF6NQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: <nil>
Dec 9 13:06:13.905292547 (93h37m13.776375592s) [seg_pos={0 0}, upload=81626]: Segment reached optimal threshold of 14 pieces.
Dec 9 13:06:13.905521141 (93h37m13.776604186s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E): protocol: expected piece hash; context canceled; EOF
Dec 9 13:06:13.905530960 (93h37m13.776613995s) [noise=true , address=45.91.116.11:4133 , piece_id=XD64PROILGAHUSDSQNI6JAIAJKGADGINUWJ5BLVBT4BP5TTABWGQ, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold
Dec 9 13:06:13.905599190 (93h37m13.776682225s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43888->145.79.227.30:4790: use of closed network connection
Dec 9 13:06:13.905618116 (93h37m13.776701161s) [noise=true , address=145.79.227.30:4790 , piece_id=5RMHVU7FKSHXO5KF7C2PCHFYVFQPWZEZCYVAFLKKD3HWTUP2KH4Q, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold
Dec 9 13:06:13.905716824 (93h37m13.776799869s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58376->64.49.13.169:4711: use of closed network connection
Dec 9 13:06:13.905727003 (93h37m13.776810038s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVUU7WE3CSMB6GSRTC6LEFPOIVT3G3GE6XHZYONZG7ZT6V2TCA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold
Dec 9 13:06:13.905954988 (93h37m13.777038043s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W3qDq8BssS8V6osW9NwFHZHMJcCRfjYP8qKwphE5SenVewWXy): protocol: expected piece hash; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; write tcp 188.166.225.170:34278->45.91.116.5:4322: use of closed network connection; EOF
Dec 9 13:06:13.905964836 (93h37m13.777047882s) [noise=true , address=45.91.116.5:4322 , piece_id=SGM4IL4M2NP5RSPO7G5YNQVYP4XG7Z3EDDNCHTQEHY7I5OORLGHA, seg_pos={0 0}, upload=81626]: Long tail cancellation due to optimal threshold
Dec 9 13:06:13.906061531 (93h37m13.777144576s) [seg_pos={0 0}, upload=81626]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 9 13:06:13.906185627 (93h37m13.777268672s) [upload=81626]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 13:06:13.906191748 (93h37m13.777274793s) [upload=81626]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 13:07:02.540166267 (93h38m2.411249312s) [upload=81626]: Done uploading segments...
Dec 9 13:07:02.540284373 (93h38m2.411367428s) [upload=81626]: Upload finished. err: metaclient: context deadline exceeded
Duration: 10.936202248s
core
uplink
us-select-1
us1-placement-20
|