Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 4.77262139s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:36:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.772654752s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 4.981743017s trace: https://jaeger.dev.storj.io/trace/1d62f0b3f1c418c0 FAIL (2025-12-09 14:36:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.98177632s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X...X............X...............X...........X........] duration: 5.615892604s trace: https://jaeger.dev.storj.io/trace/61c71b63946688f FAIL (2025-12-09 14:36:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.615930887s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [X..XXXXXXXX.XXXX..X.X.XX.....XXXX.XXX..X.....X.....X....X..X] duration: 5.001264667s trace: https://jaeger.dev.storj.io/trace/6fb100cffb0c24dc FAIL (2025-12-09 14:36:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001296147s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X....X..................................................] duration: 12.017468523s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:36:07 UTC) orbiter warning: context deadline exceeded Duration: 12.017497087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.806615797s trace: https://jaeger.dev.storj.io/trace/2d3fb38ca937a0ee FAIL (2025-12-09 14:36:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:28:25.256521148 (94h59m25.127604183s) [upload=82815]: Starting upload Dec 9 14:28:25.256577505 (94h59m25.12766055s) [upload=82815]: Done starting upload Dec 9 14:28:25.256664471 (94h59m25.127747526s) [upload=82815]: Uploading segments... Dec 9 14:28:25.256835136 (94h59m25.127918171s) [upload=82815]: Got next segment. Inline: false Dec 9 14:28:25.256919296 (94h59m25.128002331s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:28:25.256924345 (94h59m25.12800739s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:28:30.674413725 (94h59m30.54549676s) [seg_pos={0 0}, upload=82815]: Begin upload segment... Dec 9 14:28:30.674477978 (94h59m30.545561013s) [seg_pos={0 0}, upload=82815]: Done begin upload segment. Dec 9 14:28:30.674502996 (94h59m30.545586041s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.674563531 (94h59m30.545646576s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.675609904 (94h59m30.546692939s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676003132 (94h59m30.547086177s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676231287 (94h59m30.547314322s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676658069 (94h59m30.547741114s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677043223 (94h59m30.548126268s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677424559 (94h59m30.548507594s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678124031 (94h59m30.549207066s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678544131 (94h59m30.549627166s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679030797 (94h59m30.550113842s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679521632 (94h59m30.550604677s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679976518 (94h59m30.551059553s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.680584396 (94h59m30.551667441s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.681765325 (94h59m30.55284837s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.682941495 (94h59m30.55402453s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:32.391071778 (94h59m32.262154823s) [upload=82815]: Next returned nil segment Dec 9 14:28:32.391088130 (94h59m32.262171175s) [upload=82815]: Waiting for error group managing segments... Dec 9 14:28:32.690573110 (94h59m32.561656155s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.696712660 (94h59m32.567795695s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.699832612 (94h59m32.570915657s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702381074 (94h59m32.573464119s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702757842 (94h59m32.573840887s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.703091627 (94h59m32.574174682s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.716048970 (94h59m32.587132006s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.717042292 (94h59m32.588125337s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.731504031 (94h59m32.602587076s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.741912430 (94h59m32.612995465s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.760636315 (94h59m32.63171935s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.768871004 (94h59m32.639954049s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786289393 (94h59m32.657372438s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786315603 (94h59m32.657398638s) [seg_pos={0 0}, upload=82815]: Segment reached optimal threshold of 13 pieces. Dec 9 14:28:32.786602018 (94h59m32.657685063s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786611746 (94h59m32.657694801s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.786787771 (94h59m32.657870846s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786846604 (94h59m32.657929659s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787010626 (94h59m32.658093681s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.787021587 (94h59m32.658104632s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787046505 (94h59m32.65812955s) [seg_pos={0 0}, upload=82815]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:28:32.787137308 (94h59m32.658220363s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:28:32.787144471 (94h59m32.658227506s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:29:08.520820597 (95h0m8.391903682s) [upload=82815]: Done uploading segments... Dec 9 14:29:08.521201131 (95h0m8.392284186s) [upload=82815]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.806645864s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 4.93330099s trace: https://jaeger.dev.storj.io/trace/1d62f0b3f1c418c0 FAIL (2025-12-09 14:36:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.933323643s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 4.783300156s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:36:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.78334463s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 4.824153612s trace: https://jaeger.dev.storj.io/trace/1d62f0b3f1c418c0 FAIL (2025-12-09 14:35:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.824188287s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.992582348s trace: https://jaeger.dev.storj.io/trace/ab517902cf7571b FAIL (2025-12-09 14:35:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:14:03.293481138 (94h45m3.164564193s) [upload=82602]: Starting upload Dec 9 14:14:03.293630602 (94h45m3.164713667s) [upload=82602]: Done starting upload Dec 9 14:14:03.293745661 (94h45m3.164828716s) [upload=82602]: Uploading segments... Dec 9 14:14:03.293890728 (94h45m3.164973783s) [upload=82602]: Got next segment. Inline: true Dec 9 14:14:03.293897561 (94h45m3.164980616s) [upload=82602]: Waiting for error group managing segments... Dec 9 14:14:03.294051774 (94h45m3.165134809s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:14:03.294059880 (94h45m3.165142925s) [upload=82602]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 14:14:03.294063066 (94h45m3.165146111s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:15:01.261001342 (94h46m1.132084387s) [upload=82602]: Done uploading segments... Dec 9 14:15:01.261048982 (94h46m1.132132017s) [upload=82602]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.992610291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.856972615s trace: https://jaeger.dev.storj.io/trace/2d3fb38ca937a0ee FAIL (2025-12-09 14:35:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:28:25.256521148 (94h59m25.127604183s) [upload=82815]: Starting upload Dec 9 14:28:25.256577505 (94h59m25.12766055s) [upload=82815]: Done starting upload Dec 9 14:28:25.256664471 (94h59m25.127747526s) [upload=82815]: Uploading segments... Dec 9 14:28:25.256835136 (94h59m25.127918171s) [upload=82815]: Got next segment. Inline: false Dec 9 14:28:25.256919296 (94h59m25.128002331s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:28:25.256924345 (94h59m25.12800739s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:28:30.674413725 (94h59m30.54549676s) [seg_pos={0 0}, upload=82815]: Begin upload segment... Dec 9 14:28:30.674477978 (94h59m30.545561013s) [seg_pos={0 0}, upload=82815]: Done begin upload segment. Dec 9 14:28:30.674502996 (94h59m30.545586041s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.674563531 (94h59m30.545646576s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.675609904 (94h59m30.546692939s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676003132 (94h59m30.547086177s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676231287 (94h59m30.547314322s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676658069 (94h59m30.547741114s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677043223 (94h59m30.548126268s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677424559 (94h59m30.548507594s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678124031 (94h59m30.549207066s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678544131 (94h59m30.549627166s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679030797 (94h59m30.550113842s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679521632 (94h59m30.550604677s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679976518 (94h59m30.551059553s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.680584396 (94h59m30.551667441s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.681765325 (94h59m30.55284837s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.682941495 (94h59m30.55402453s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:32.391071778 (94h59m32.262154823s) [upload=82815]: Next returned nil segment Dec 9 14:28:32.391088130 (94h59m32.262171175s) [upload=82815]: Waiting for error group managing segments... Dec 9 14:28:32.690573110 (94h59m32.561656155s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.696712660 (94h59m32.567795695s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.699832612 (94h59m32.570915657s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702381074 (94h59m32.573464119s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702757842 (94h59m32.573840887s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.703091627 (94h59m32.574174682s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.716048970 (94h59m32.587132006s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.717042292 (94h59m32.588125337s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.731504031 (94h59m32.602587076s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.741912430 (94h59m32.612995465s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.760636315 (94h59m32.63171935s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.768871004 (94h59m32.639954049s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786289393 (94h59m32.657372438s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786315603 (94h59m32.657398638s) [seg_pos={0 0}, upload=82815]: Segment reached optimal threshold of 13 pieces. Dec 9 14:28:32.786602018 (94h59m32.657685063s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786611746 (94h59m32.657694801s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.786787771 (94h59m32.657870846s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786846604 (94h59m32.657929659s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787010626 (94h59m32.658093681s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.787021587 (94h59m32.658104632s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787046505 (94h59m32.65812955s) [seg_pos={0 0}, upload=82815]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:28:32.787137308 (94h59m32.658220363s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:28:32.787144471 (94h59m32.658227506s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:29:08.520820597 (95h0m8.391903682s) [upload=82815]: Done uploading segments... Dec 9 14:29:08.521201131 (95h0m8.392284186s) [upload=82815]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.856999647s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [XX..XXXXXXXX.XXXX..X.X.XX.....XXXX.XXX..X.....X.....X....X..] duration: 3.767712754s trace: https://jaeger.dev.storj.io/trace/175fd463c8dc3a74 FAIL (2025-12-09 14:35:31 UTC) orbiter warning: context deadline exceeded Duration: 3.767738674s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X....X.................................................] duration: 11.32910478s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:35:26 UTC) orbiter warning: context deadline exceeded Duration: 11.329124178s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [........X...X.X....X.....X........X..XX..XX...X....X........] duration: 22.324630384s trace: https://jaeger.dev.storj.io/trace/5ee6b0f07ada006 FAIL (2025-12-09 14:35:23 UTC) orbiter warning: context deadline exceeded Duration: 22.324656132s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X...X............X...............X...........X.......] duration: 5.842241057s trace: https://jaeger.dev.storj.io/trace/61c71b63946688f FAIL (2025-12-09 14:35:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.842271114s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X....X................................................] duration: 26.945823107s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:35:14 UTC) orbiter warning: context deadline exceeded Duration: 26.945851229s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [XXX..XXXXXXXX.XXXX..X.X.XX.....XXXX.XXX..X.....X.....X....X.] duration: 4.950544496s trace: https://jaeger.dev.storj.io/trace/175fd463c8dc3a74 FAIL (2025-12-09 14:35:00 UTC) orbiter warning: context deadline exceeded Duration: 4.950571166s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X...X............X...............X...........X......] duration: 5.912539047s trace: https://jaeger.dev.storj.io/trace/61c71b63946688f FAIL (2025-12-09 14:34:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.912566298s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 4.699475719s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:34:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.69950254s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 4.81196653s trace: https://jaeger.dev.storj.io/trace/1d62f0b3f1c418c0 FAIL (2025-12-09 14:34:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.811996206s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.........X...X.X....X.....X........X..XX..XX...X....X.......] duration: 23.18114129s trace: https://jaeger.dev.storj.io/trace/5ee6b0f07ada006 FAIL (2025-12-09 14:34:33 UTC) orbiter warning: context deadline exceeded Duration: 23.181187729s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.990190616s trace: https://jaeger.dev.storj.io/trace/2d3fb38ca937a0ee FAIL (2025-12-09 14:34:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:28:25.256521148 (94h59m25.127604183s) [upload=82815]: Starting upload Dec 9 14:28:25.256577505 (94h59m25.12766055s) [upload=82815]: Done starting upload Dec 9 14:28:25.256664471 (94h59m25.127747526s) [upload=82815]: Uploading segments... Dec 9 14:28:25.256835136 (94h59m25.127918171s) [upload=82815]: Got next segment. Inline: false Dec 9 14:28:25.256919296 (94h59m25.128002331s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:28:25.256924345 (94h59m25.12800739s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:28:30.674413725 (94h59m30.54549676s) [seg_pos={0 0}, upload=82815]: Begin upload segment... Dec 9 14:28:30.674477978 (94h59m30.545561013s) [seg_pos={0 0}, upload=82815]: Done begin upload segment. Dec 9 14:28:30.674502996 (94h59m30.545586041s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.674563531 (94h59m30.545646576s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.675609904 (94h59m30.546692939s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676003132 (94h59m30.547086177s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676231287 (94h59m30.547314322s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676658069 (94h59m30.547741114s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677043223 (94h59m30.548126268s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677424559 (94h59m30.548507594s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678124031 (94h59m30.549207066s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678544131 (94h59m30.549627166s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679030797 (94h59m30.550113842s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679521632 (94h59m30.550604677s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679976518 (94h59m30.551059553s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.680584396 (94h59m30.551667441s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.681765325 (94h59m30.55284837s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.682941495 (94h59m30.55402453s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:32.391071778 (94h59m32.262154823s) [upload=82815]: Next returned nil segment Dec 9 14:28:32.391088130 (94h59m32.262171175s) [upload=82815]: Waiting for error group managing segments... Dec 9 14:28:32.690573110 (94h59m32.561656155s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.696712660 (94h59m32.567795695s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.699832612 (94h59m32.570915657s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702381074 (94h59m32.573464119s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702757842 (94h59m32.573840887s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.703091627 (94h59m32.574174682s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.716048970 (94h59m32.587132006s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.717042292 (94h59m32.588125337s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.731504031 (94h59m32.602587076s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.741912430 (94h59m32.612995465s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.760636315 (94h59m32.63171935s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.768871004 (94h59m32.639954049s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786289393 (94h59m32.657372438s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786315603 (94h59m32.657398638s) [seg_pos={0 0}, upload=82815]: Segment reached optimal threshold of 13 pieces. Dec 9 14:28:32.786602018 (94h59m32.657685063s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786611746 (94h59m32.657694801s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.786787771 (94h59m32.657870846s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786846604 (94h59m32.657929659s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787010626 (94h59m32.658093681s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.787021587 (94h59m32.658104632s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787046505 (94h59m32.65812955s) [seg_pos={0 0}, upload=82815]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:28:32.787137308 (94h59m32.658220363s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:28:32.787144471 (94h59m32.658227506s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:29:08.520820597 (95h0m8.391903682s) [upload=82815]: Done uploading segments... Dec 9 14:29:08.521201131 (95h0m8.392284186s) [upload=82815]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.990221004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.092595684s trace: https://jaeger.dev.storj.io/trace/ab517902cf7571b FAIL (2025-12-09 14:34:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:14:03.293481138 (94h45m3.164564193s) [upload=82602]: Starting upload Dec 9 14:14:03.293630602 (94h45m3.164713667s) [upload=82602]: Done starting upload Dec 9 14:14:03.293745661 (94h45m3.164828716s) [upload=82602]: Uploading segments... Dec 9 14:14:03.293890728 (94h45m3.164973783s) [upload=82602]: Got next segment. Inline: true Dec 9 14:14:03.293897561 (94h45m3.164980616s) [upload=82602]: Waiting for error group managing segments... Dec 9 14:14:03.294051774 (94h45m3.165134809s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:14:03.294059880 (94h45m3.165142925s) [upload=82602]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 14:14:03.294063066 (94h45m3.165146111s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:15:01.261001342 (94h46m1.132084387s) [upload=82602]: Done uploading segments... Dec 9 14:15:01.261048982 (94h46m1.132132017s) [upload=82602]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.09261465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 4.688523934s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:34:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.688555465s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.911226457s trace: https://jaeger.dev.storj.io/trace/2d3fb38ca937a0ee FAIL (2025-12-09 14:34:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:28:25.256521148 (94h59m25.127604183s) [upload=82815]: Starting upload Dec 9 14:28:25.256577505 (94h59m25.12766055s) [upload=82815]: Done starting upload Dec 9 14:28:25.256664471 (94h59m25.127747526s) [upload=82815]: Uploading segments... Dec 9 14:28:25.256835136 (94h59m25.127918171s) [upload=82815]: Got next segment. Inline: false Dec 9 14:28:25.256919296 (94h59m25.128002331s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:28:25.256924345 (94h59m25.12800739s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:28:30.674413725 (94h59m30.54549676s) [seg_pos={0 0}, upload=82815]: Begin upload segment... Dec 9 14:28:30.674477978 (94h59m30.545561013s) [seg_pos={0 0}, upload=82815]: Done begin upload segment. Dec 9 14:28:30.674502996 (94h59m30.545586041s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.674563531 (94h59m30.545646576s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.675609904 (94h59m30.546692939s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676003132 (94h59m30.547086177s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676231287 (94h59m30.547314322s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676658069 (94h59m30.547741114s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677043223 (94h59m30.548126268s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677424559 (94h59m30.548507594s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678124031 (94h59m30.549207066s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678544131 (94h59m30.549627166s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679030797 (94h59m30.550113842s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679521632 (94h59m30.550604677s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679976518 (94h59m30.551059553s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.680584396 (94h59m30.551667441s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.681765325 (94h59m30.55284837s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.682941495 (94h59m30.55402453s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:32.391071778 (94h59m32.262154823s) [upload=82815]: Next returned nil segment Dec 9 14:28:32.391088130 (94h59m32.262171175s) [upload=82815]: Waiting for error group managing segments... Dec 9 14:28:32.690573110 (94h59m32.561656155s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.696712660 (94h59m32.567795695s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.699832612 (94h59m32.570915657s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702381074 (94h59m32.573464119s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702757842 (94h59m32.573840887s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.703091627 (94h59m32.574174682s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.716048970 (94h59m32.587132006s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.717042292 (94h59m32.588125337s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.731504031 (94h59m32.602587076s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.741912430 (94h59m32.612995465s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.760636315 (94h59m32.63171935s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.768871004 (94h59m32.639954049s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786289393 (94h59m32.657372438s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786315603 (94h59m32.657398638s) [seg_pos={0 0}, upload=82815]: Segment reached optimal threshold of 13 pieces. Dec 9 14:28:32.786602018 (94h59m32.657685063s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786611746 (94h59m32.657694801s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.786787771 (94h59m32.657870846s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786846604 (94h59m32.657929659s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787010626 (94h59m32.658093681s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.787021587 (94h59m32.658104632s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787046505 (94h59m32.65812955s) [seg_pos={0 0}, upload=82815]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:28:32.787137308 (94h59m32.658220363s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:28:32.787144471 (94h59m32.658227506s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:29:08.520820597 (95h0m8.391903682s) [upload=82815]: Done uploading segments... Dec 9 14:29:08.521201131 (95h0m8.392284186s) [upload=82815]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.911263567s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X...X............X...............X...........X.....] duration: 5.808957384s trace: https://jaeger.dev.storj.io/trace/61c71b63946688f FAIL (2025-12-09 14:33:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.808984578s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...................................................X.......] duration: 4.883129241s trace: https://jaeger.dev.storj.io/trace/1d62f0b3f1c418c0 FAIL (2025-12-09 14:33:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.883163638s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........X...X.X....X.....X........X..XX..XX...X....X......] duration: 24.779390675s trace: https://jaeger.dev.storj.io/trace/5ee6b0f07ada006 FAIL (2025-12-09 14:33:51 UTC) orbiter warning: context deadline exceeded Duration: 24.779420332s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [XXXX..XXXXXXXX.XXXX..X.X.XX.....XXXX.XXX..X.....X.....X....X] duration: 5.001993934s trace: https://jaeger.dev.storj.io/trace/175fd463c8dc3a74 FAIL (2025-12-09 14:33:51 UTC) orbiter warning: context deadline exceeded Duration: 5.002020254s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X....X...............................................] duration: 6.442665253s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:33:51 UTC) orbiter warning: context deadline exceeded Duration: 6.442697865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 4.793459032s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:33:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.793490303s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X...X............X...............X...........X....] duration: 5.849762516s trace: https://jaeger.dev.storj.io/trace/61c71b63946688f FAIL (2025-12-09 14:33:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.849791471s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...................................................X......] duration: 4.953816886s trace: https://jaeger.dev.storj.io/trace/1d62f0b3f1c418c0 FAIL (2025-12-09 14:33:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.953847493s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [XXXXX..XXXXXXXX.XXXX..X.X.XX.....XXXX.XXX..X.....X.....X....] duration: 5.001031352s trace: https://jaeger.dev.storj.io/trace/70269f78a3bd5209 FAIL (2025-12-09 14:33:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001211895s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X....X..............................................] duration: 9.815922327s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:33:15 UTC) orbiter warning: context deadline exceeded Duration: 9.81598177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.055529383s trace: https://jaeger.dev.storj.io/trace/ab517902cf7571b FAIL (2025-12-09 14:33:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:14:03.293481138 (94h45m3.164564193s) [upload=82602]: Starting upload Dec 9 14:14:03.293630602 (94h45m3.164713667s) [upload=82602]: Done starting upload Dec 9 14:14:03.293745661 (94h45m3.164828716s) [upload=82602]: Uploading segments... Dec 9 14:14:03.293890728 (94h45m3.164973783s) [upload=82602]: Got next segment. Inline: true Dec 9 14:14:03.293897561 (94h45m3.164980616s) [upload=82602]: Waiting for error group managing segments... Dec 9 14:14:03.294051774 (94h45m3.165134809s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:14:03.294059880 (94h45m3.165142925s) [upload=82602]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 14:14:03.294063066 (94h45m3.165146111s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:15:01.261001342 (94h46m1.132084387s) [upload=82602]: Done uploading segments... Dec 9 14:15:01.261048982 (94h46m1.132132017s) [upload=82602]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.055569881s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...........X...X.X....X.....X........X..XX..XX...X....X.....] duration: 25.08450521s trace: https://jaeger.dev.storj.io/trace/5ee6b0f07ada006 FAIL (2025-12-09 14:32:57 UTC) orbiter warning: context deadline exceeded Duration: 25.084556026s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.781994776s trace: https://jaeger.dev.storj.io/trace/2d3fb38ca937a0ee FAIL (2025-12-09 14:32:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:28:25.256521148 (94h59m25.127604183s) [upload=82815]: Starting upload Dec 9 14:28:25.256577505 (94h59m25.12766055s) [upload=82815]: Done starting upload Dec 9 14:28:25.256664471 (94h59m25.127747526s) [upload=82815]: Uploading segments... Dec 9 14:28:25.256835136 (94h59m25.127918171s) [upload=82815]: Got next segment. Inline: false Dec 9 14:28:25.256919296 (94h59m25.128002331s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:28:25.256924345 (94h59m25.12800739s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:28:30.674413725 (94h59m30.54549676s) [seg_pos={0 0}, upload=82815]: Begin upload segment... Dec 9 14:28:30.674477978 (94h59m30.545561013s) [seg_pos={0 0}, upload=82815]: Done begin upload segment. Dec 9 14:28:30.674502996 (94h59m30.545586041s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.674563531 (94h59m30.545646576s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.675609904 (94h59m30.546692939s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676003132 (94h59m30.547086177s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676231287 (94h59m30.547314322s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676658069 (94h59m30.547741114s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677043223 (94h59m30.548126268s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677424559 (94h59m30.548507594s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678124031 (94h59m30.549207066s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678544131 (94h59m30.549627166s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679030797 (94h59m30.550113842s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679521632 (94h59m30.550604677s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679976518 (94h59m30.551059553s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.680584396 (94h59m30.551667441s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.681765325 (94h59m30.55284837s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.682941495 (94h59m30.55402453s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:32.391071778 (94h59m32.262154823s) [upload=82815]: Next returned nil segment Dec 9 14:28:32.391088130 (94h59m32.262171175s) [upload=82815]: Waiting for error group managing segments... Dec 9 14:28:32.690573110 (94h59m32.561656155s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.696712660 (94h59m32.567795695s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.699832612 (94h59m32.570915657s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702381074 (94h59m32.573464119s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702757842 (94h59m32.573840887s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.703091627 (94h59m32.574174682s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.716048970 (94h59m32.587132006s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.717042292 (94h59m32.588125337s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.731504031 (94h59m32.602587076s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.741912430 (94h59m32.612995465s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.760636315 (94h59m32.63171935s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.768871004 (94h59m32.639954049s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786289393 (94h59m32.657372438s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786315603 (94h59m32.657398638s) [seg_pos={0 0}, upload=82815]: Segment reached optimal threshold of 13 pieces. Dec 9 14:28:32.786602018 (94h59m32.657685063s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786611746 (94h59m32.657694801s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.786787771 (94h59m32.657870846s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786846604 (94h59m32.657929659s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787010626 (94h59m32.658093681s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.787021587 (94h59m32.658104632s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787046505 (94h59m32.65812955s) [seg_pos={0 0}, upload=82815]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:28:32.787137308 (94h59m32.658220363s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:28:32.787144471 (94h59m32.658227506s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:29:08.520820597 (95h0m8.391903682s) [upload=82815]: Done uploading segments... Dec 9 14:29:08.521201131 (95h0m8.392284186s) [upload=82815]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.782020975s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..........X...X............X...............X...........X...] duration: 5.8412082s trace: https://jaeger.dev.storj.io/trace/61c71b63946688f FAIL (2025-12-09 14:32:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.841236805s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X....X.............................................] duration: 8.7107824s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:32:52 UTC) orbiter warning: context deadline exceeded Duration: 8.71080887s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.006554084s trace: https://jaeger.dev.storj.io/trace/ab517902cf7571b FAIL (2025-12-09 14:32:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:14:03.293481138 (94h45m3.164564193s) [upload=82602]: Starting upload Dec 9 14:14:03.293630602 (94h45m3.164713667s) [upload=82602]: Done starting upload Dec 9 14:14:03.293745661 (94h45m3.164828716s) [upload=82602]: Uploading segments... Dec 9 14:14:03.293890728 (94h45m3.164973783s) [upload=82602]: Got next segment. Inline: true Dec 9 14:14:03.293897561 (94h45m3.164980616s) [upload=82602]: Waiting for error group managing segments... Dec 9 14:14:03.294051774 (94h45m3.165134809s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:14:03.294059880 (94h45m3.165142925s) [upload=82602]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 14:14:03.294063066 (94h45m3.165146111s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:15:01.261001342 (94h46m1.132084387s) [upload=82602]: Done uploading segments... Dec 9 14:15:01.261048982 (94h46m1.132132017s) [upload=82602]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.006582509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 4.714196891s trace: https://jaeger.dev.storj.io/trace/55833ed0d1a4f5c7 FAIL (2025-12-09 14:32:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.888756976 (94h26m18.759840011s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890128928 (94h26m18.761211963s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890515013 (94h26m18.761598048s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.890833250 (94h26m18.761916285s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891303795 (94h26m18.76238684s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.891696983 (94h26m18.762780028s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892039806 (94h26m18.763122841s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.892451160 (94h26m18.763534205s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895174816 (94h26m18.766257861s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:18.895700677 (94h26m18.766783712s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Uploading piece... Dec 9 13:55:20.096396486 (94h26m19.967479521s) [upload=82327]: Next returned nil segment Dec 9 13:55:20.096410893 (94h26m19.967493928s) [upload=82327]: Waiting for error group managing segments... Dec 9 13:55:20.661493357 (94h26m20.532576402s) [noise=true , address=185.167.204.206:4048 , piece_id=4RZ7RD6JDLKT2ZQEREYWWJM7QCAZXM3P2KQJFSXH56E7PNJLMY2Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.662326092 (94h26m20.533409137s) [noise=true , address=185.167.204.206:4060 , piece_id=CUTLMZYWD3UV5LV6GBCAMXUQQJ25CUHTVB5XNMQ2EKZEOOG4SZTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.671455297 (94h26m20.542538342s) [noise=true , address=185.167.204.206:4057 , piece_id=X4RRHTHRB44OQCWUPFNK3LM6BCN6RAHP3T2XJLEBTCH7YIDBXXAA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.673985184 (94h26m20.545068259s) [noise=true , address=81.171.14.70:4035 , piece_id=YYMWRJBTHXAX5VL6SUYBO6KMQ2SD25ZME3ZVLEX7Q6WHSRLAODUA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.689475531 (94h26m20.560558576s) [noise=true , address=185.167.204.205:4015 , piece_id=MD6GVL7PXEW5POZUFEFMJ2FEDXJ4F3CCB2VGIUE6EV32N65OF4ZA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.692722855 (94h26m20.5638059s) [noise=true , address=185.167.204.205:4029 , piece_id=HLOAXOFQA3N2ZQFUNJI4BSH3ZTZKP3AXT2IZRA36L2QNMPGGIPJA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.707950632 (94h26m20.579033667s) [noise=true , address=185.167.204.206:4053 , piece_id=HZOMLUDVZYQ7QENJS6OHOXPGRW2WLZPQHWLCH734PGCTMUU7EHGQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.709195873 (94h26m20.580278918s) [noise=true , address=81.171.14.70:4046 , piece_id=6JJXIEIXQHLPSHSY5DN3VB4WYQW5LY5WOL6PISOFJ4I67ORZ5ECA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.714610222 (94h26m20.585693267s) [noise=true , address=185.167.204.206:4065 , piece_id=UWZUGSNUACUVBNXN6E44QE75ULM23RCHWANCNMNPE5PAALES277A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.719569186 (94h26m20.590652231s) [noise=true , address=185.167.204.206:4055 , piece_id=EYOBIGLZRXYG5T2YSHI6D5GGZOE3OROR4NJQ7W7PFZGSNXGGGZKQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.725745656 (94h26m20.596828701s) [noise=true , address=185.167.204.205:4026 , piece_id=LOLW2DYACI5Y6J3HR3VZIGZOMBELUKNOVTGUMP6R22TBK4DJRFHA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.742603277 (94h26m20.613686322s) [noise=true , address=185.167.204.206:4037 , piece_id=CRHCSOLMZOOUVL5ON4ZLXLMF6QQU5MCVHNBKXYZUXZ7CXZYU6KLA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.751494787 (94h26m20.622577822s) [noise=true , address=185.167.204.206:4061 , piece_id=NAHSMEJRVXEMKSGCJTGQDJ657T5PZUYYJXCPPCZV5C3SPPU2EV3A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.808800229 (94h26m20.679883264s) [noise=true , address=81.171.14.70:4032 , piece_id=PCOM4TRHYZWLUOUBZZM3WLDC7M2RQWA6O7TGL22RWQC4C3EHPG7A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.809236991 (94h26m20.680320036s) [noise=true , address=81.171.14.70:4025 , piece_id=LKRZ5544ETEJQMDAFCBBE3YQENQMYJ352PSBUCYCBHSBSRHFJUWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816163029 (94h26m20.687246064s) [noise=true , address=81.171.14.70:4028 , piece_id=2FKFFWS4ZMFAHMMQPRU4BW4OMZXTR5HJ3DJ54ZHLA3QAUK2AKERA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:20.816671367 (94h26m20.687754412s) [noise=true , address=81.171.14.70:4034 , piece_id=I4OVL5OITRXCVZLUPJIG5X3JGJPHMUVPG4BBU3T35HI7APV2TIWQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.268225529 (94h26m21.139308564s) [noise=true , address=81.171.14.70:4030 , piece_id=37VYH6EPMN3ONMDSVMNEB3HCSOGVINDKKKDTALHWWF3Z3TXSIDBQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.335251881 (94h26m21.206334926s) [noise=true , address=89.149.201.147:4020 , piece_id=MZC6BJL76C4VBDCMJSCTEQXKD26IGKMIDWXAAKYHNVX3SIUHLRTQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.349903661 (94h26m21.220986706s) [noise=true , address=89.149.201.147:4022 , piece_id=LA7F73VPFKLAL3PIR2WZVPFK2NHGN7RCB3GXNTHXBUNBT3MLAFMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.399372740 (94h26m21.270455785s) [noise=true , address=89.149.201.147:4009 , piece_id=CYSG3LUAFDHI4L3K5XWKJV4T4Z2UAILO7IOLWEWPRRICO3FHTYJQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.414140341 (94h26m21.285223386s) [noise=true , address=89.149.201.147:4005 , piece_id=5ABM3E4ROULAPFGVXOMYQ7YB4VLN7DYDRXYTLBDDMLORETS4P5BQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.423538717 (94h26m21.294621762s) [noise=true , address=89.149.201.147:4014 , piece_id=3ZQQSGSJXKWQ5Q62BU4VCTOWMIBM6VHPPW22SJSHFHBZ43TRIZIQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.436510208 (94h26m21.307593243s) [noise=true , address=89.149.201.147:4016 , piece_id=L7QHMXZRMJRAZTS3P5STCUOAWTANQ34YAO5UFQPEZLP2BLHZWOHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.466064375 (94h26m21.33714742s) [noise=true , address=89.149.201.147:4010 , piece_id=JQDSJCYWFBDPGIFKJIGX6Z6C4ER2DZGUILVBY6SRGTQSK4Z6EQUQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491513065 (94h26m21.3625961s) [noise=true , address=89.149.201.147:4012 , piece_id=KMTIUM4XLXYV42EZ3BDRSK2T4IJSJPCO4OBHPBXIKNJJUEV5P2OQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Dec 9 13:55:21.491533172 (94h26m21.362616217s) [seg_pos={0 0}, upload=82327]: Segment reached optimal threshold of 26 pieces. Dec 9 13:55:21.491858713 (94h26m21.362941748s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491865335 (94h26m21.36294838s) [noise=true , address=81.171.14.70:4026 , piece_id=YFEC7SKBP2PHUKMB6QZ7VVTNLF6XYGY4NGSW6MDDH4IUV7PEK4ZQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.491984081 (94h26m21.363067116s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 9 13:55:21.491989641 (94h26m21.363072676s) [noise=true , address=89.149.201.147:4018 , piece_id=2VIYBM2CYDAZBTJ2WBEB47LIMWJH4MLUR2B2W3ZE4VCBWHPLF5RQ, seg_pos={0 0}, upload=82327]: Long tail cancellation due to optimal threshold Dec 9 13:55:21.492007145 (94h26m21.36309018s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 13:55:21.492082318 (94h26m21.363165363s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 13:55:21.492088420 (94h26m21.363171465s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 13:55:21.731576485 (94h26m21.60265952s) [upload=82327]: Done uploading segments... Dec 9 13:55:21.731594659 (94h26m21.602677704s) [upload=82327]: Upload finished. err: <nil> Duration: 4.714236528s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.853820092s trace: https://jaeger.dev.storj.io/trace/2d3fb38ca937a0ee FAIL (2025-12-09 14:32:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:28:25.256521148 (94h59m25.127604183s) [upload=82815]: Starting upload Dec 9 14:28:25.256577505 (94h59m25.12766055s) [upload=82815]: Done starting upload Dec 9 14:28:25.256664471 (94h59m25.127747526s) [upload=82815]: Uploading segments... Dec 9 14:28:25.256835136 (94h59m25.127918171s) [upload=82815]: Got next segment. Inline: false Dec 9 14:28:25.256919296 (94h59m25.128002331s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:28:25.256924345 (94h59m25.12800739s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:28:30.674413725 (94h59m30.54549676s) [seg_pos={0 0}, upload=82815]: Begin upload segment... Dec 9 14:28:30.674477978 (94h59m30.545561013s) [seg_pos={0 0}, upload=82815]: Done begin upload segment. Dec 9 14:28:30.674502996 (94h59m30.545586041s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.674563531 (94h59m30.545646576s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.675609904 (94h59m30.546692939s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676003132 (94h59m30.547086177s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676231287 (94h59m30.547314322s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676658069 (94h59m30.547741114s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677043223 (94h59m30.548126268s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677424559 (94h59m30.548507594s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678124031 (94h59m30.549207066s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678544131 (94h59m30.549627166s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679030797 (94h59m30.550113842s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679521632 (94h59m30.550604677s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679976518 (94h59m30.551059553s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.680584396 (94h59m30.551667441s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.681765325 (94h59m30.55284837s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.682941495 (94h59m30.55402453s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:32.391071778 (94h59m32.262154823s) [upload=82815]: Next returned nil segment Dec 9 14:28:32.391088130 (94h59m32.262171175s) [upload=82815]: Waiting for error group managing segments... Dec 9 14:28:32.690573110 (94h59m32.561656155s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.696712660 (94h59m32.567795695s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.699832612 (94h59m32.570915657s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702381074 (94h59m32.573464119s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702757842 (94h59m32.573840887s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.703091627 (94h59m32.574174682s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.716048970 (94h59m32.587132006s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.717042292 (94h59m32.588125337s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.731504031 (94h59m32.602587076s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.741912430 (94h59m32.612995465s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.760636315 (94h59m32.63171935s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.768871004 (94h59m32.639954049s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786289393 (94h59m32.657372438s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786315603 (94h59m32.657398638s) [seg_pos={0 0}, upload=82815]: Segment reached optimal threshold of 13 pieces. Dec 9 14:28:32.786602018 (94h59m32.657685063s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786611746 (94h59m32.657694801s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.786787771 (94h59m32.657870846s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786846604 (94h59m32.657929659s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787010626 (94h59m32.658093681s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.787021587 (94h59m32.658104632s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787046505 (94h59m32.65812955s) [seg_pos={0 0}, upload=82815]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:28:32.787137308 (94h59m32.658220363s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:28:32.787144471 (94h59m32.658227506s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:29:08.520820597 (95h0m8.391903682s) [upload=82815]: Done uploading segments... Dec 9 14:29:08.521201131 (95h0m8.392284186s) [upload=82815]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.85384974s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [.XXXXX..XXXXXXXX.XXXX..X.X.XX.....XXXX.XXX..X.....X.....X...] duration: 4.854025245s trace: https://jaeger.dev.storj.io/trace/70269f78a3bd5209 FAIL (2025-12-09 14:32:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.85405976s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...................................................X.....] duration: 4.84992836s trace: https://jaeger.dev.storj.io/trace/1d62f0b3f1c418c0 FAIL (2025-12-09 14:32:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.849958087s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [............X...X.X....X.....X........X..XX..XX...X....X....] duration: 7.79427289s trace: https://jaeger.dev.storj.io/trace/5ee6b0f07ada006 FAIL (2025-12-09 14:32:13 UTC) orbiter warning: context deadline exceeded Duration: 7.794293339s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.883107114s trace: https://jaeger.dev.storj.io/trace/2d3fb38ca937a0ee FAIL (2025-12-09 14:32:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:28:25.256521148 (94h59m25.127604183s) [upload=82815]: Starting upload Dec 9 14:28:25.256577505 (94h59m25.12766055s) [upload=82815]: Done starting upload Dec 9 14:28:25.256664471 (94h59m25.127747526s) [upload=82815]: Uploading segments... Dec 9 14:28:25.256835136 (94h59m25.127918171s) [upload=82815]: Got next segment. Inline: false Dec 9 14:28:25.256919296 (94h59m25.128002331s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:28:25.256924345 (94h59m25.12800739s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:28:30.674413725 (94h59m30.54549676s) [seg_pos={0 0}, upload=82815]: Begin upload segment... Dec 9 14:28:30.674477978 (94h59m30.545561013s) [seg_pos={0 0}, upload=82815]: Done begin upload segment. Dec 9 14:28:30.674502996 (94h59m30.545586041s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.674563531 (94h59m30.545646576s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.675609904 (94h59m30.546692939s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676003132 (94h59m30.547086177s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676231287 (94h59m30.547314322s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.676658069 (94h59m30.547741114s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677043223 (94h59m30.548126268s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.677424559 (94h59m30.548507594s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678124031 (94h59m30.549207066s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.678544131 (94h59m30.549627166s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679030797 (94h59m30.550113842s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679521632 (94h59m30.550604677s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.679976518 (94h59m30.551059553s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.680584396 (94h59m30.551667441s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.681765325 (94h59m30.55284837s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:30.682941495 (94h59m30.55402453s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Uploading piece... Dec 9 14:28:32.391071778 (94h59m32.262154823s) [upload=82815]: Next returned nil segment Dec 9 14:28:32.391088130 (94h59m32.262171175s) [upload=82815]: Waiting for error group managing segments... Dec 9 14:28:32.690573110 (94h59m32.561656155s) [noise=true , address=145.79.227.26:4655 , piece_id=NFFFONC4WD5SQXZPCI4PAZSPZH7SVK3X45GRLOSG6DRKI2RKJM3Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.696712660 (94h59m32.567795695s) [noise=true , address=145.79.227.22:4510 , piece_id=K7YM3YNQBKF2URTUPGWMM6QNHE4HSBXUJ64RGJIDX7MQWVHBA5IA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.699832612 (94h59m32.570915657s) [noise=true , address=145.79.227.39:4013 , piece_id=UC2CZ5ZF745UJF5WX45E4QD4ZCTCJLWAWZN5UOZSSGR4YEYUB2KQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702381074 (94h59m32.573464119s) [noise=true , address=145.79.227.33:4869 , piece_id=36WS6KWNX6TC3FWNUP3JL55JYSKYQ3FGVJJGFRZFBO3EFTRZ2M2Q, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.702757842 (94h59m32.573840887s) [noise=true , address=145.79.227.38:4979 , piece_id=PSF6IX4WRQKCOO34MU5PRAMCO2RREZFX7VPIVVXXR4J4FWSVVWIA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.703091627 (94h59m32.574174682s) [noise=true , address=145.79.227.22:4509 , piece_id=4KQU7FGZV7GLWOK3FZWTSESM2LMOZMHVISICRHMGFMS2XZ3MW73A, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.716048970 (94h59m32.587132006s) [noise=true , address=145.79.227.23:4546 , piece_id=UJR3JLHFCAUMN53YPEHBATY5RQ5KIRD3TIO3VM6IQX3WLRCVN4ZQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.717042292 (94h59m32.588125337s) [noise=true , address=145.79.227.32:4834 , piece_id=UMN5PXSG56MEX4HGNSYHTKPIT423CP7XI2BSQGTS6ZPO7RWSRMRA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.731504031 (94h59m32.602587076s) [noise=true , address=145.79.227.32:4835 , piece_id=EG3V6R23TZE6N32KTKIT5DO3Y7WYHPUALOAH5DDRTHYQD7QJPXCA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.741912430 (94h59m32.612995465s) [noise=true , address=145.79.227.28:4726 , piece_id=IC56P4BNTIIPIQXGSCUTLBUO57BTGVYT76T46VZPDP3POARI57HA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.760636315 (94h59m32.63171935s) [noise=true , address=45.91.116.12:4226 , piece_id=ONMMGV6624GJUWWPUVBGHAZ245Q6EBXSBRWUH4UCE4UFVEFNOCAQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.768871004 (94h59m32.639954049s) [noise=true , address=45.91.116.8:4045 , piece_id=ZLZWMUPU2DC6R6MPUIRXS4AWN26KSB3WXJY7AHMXVCYKDYOYHHJA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786289393 (94h59m32.657372438s) [noise=true , address=145.79.227.43:4159 , piece_id=SYFNRB53EMOCWFJSFTZCAPQZQ2HZCSG4XMJY764MU7ROVJNA2EDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: <nil> Dec 9 14:28:32.786315603 (94h59m32.657398638s) [seg_pos={0 0}, upload=82815]: Segment reached optimal threshold of 13 pieces. Dec 9 14:28:32.786602018 (94h59m32.657685063s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786611746 (94h59m32.657694801s) [noise=true , address=45.91.116.12:4227 , piece_id=AZ6W7YPLIMY5SDIBSUI466PMUS65CGOAWVYYGO3X4LB5RNJT4LDA, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.786787771 (94h59m32.657870846s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.786846604 (94h59m32.657929659s) [noise=true , address=145.79.227.41:4085 , piece_id=XJGDFWMPD4TCMN2N4KVI6R4W67FWEBF64X6FL6F2GGRBFWWYDEDQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787010626 (94h59m32.658093681s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 9 14:28:32.787021587 (94h59m32.658104632s) [noise=true , address=145.79.227.33:4870 , piece_id=VS6HA27XFGTWS3ANWONBU3GKOU23VRQTD5TAM23NBBCSQ35MMRSQ, seg_pos={0 0}, upload=82815]: Long tail cancellation due to optimal threshold Dec 9 14:28:32.787046505 (94h59m32.65812955s) [seg_pos={0 0}, upload=82815]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:28:32.787137308 (94h59m32.658220363s) [upload=82815]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:28:32.787144471 (94h59m32.658227506s) [upload=82815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:29:08.520820597 (95h0m8.391903682s) [upload=82815]: Done uploading segments... Dec 9 14:29:08.521201131 (95h0m8.392284186s) [upload=82815]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.883552071s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..........X...X............X...............X...........X..] duration: 5.915291298s trace: https://jaeger.dev.storj.io/trace/61c71b63946688f FAIL (2025-12-09 14:32:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.915323308s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X....X............................................] duration: 11.267325314s trace: https://jaeger.dev.storj.io/trace/7085a484f4385819 FAIL (2025-12-09 14:31:58 UTC) orbiter warning: context deadline exceeded Duration: 11.267378556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 24.410280369s trace: https://jaeger.dev.storj.io/trace/ab517902cf7571b FAIL (2025-12-09 14:31:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:14:03.293481138 (94h45m3.164564193s) [upload=82602]: Starting upload Dec 9 14:14:03.293630602 (94h45m3.164713667s) [upload=82602]: Done starting upload Dec 9 14:14:03.293745661 (94h45m3.164828716s) [upload=82602]: Uploading segments... Dec 9 14:14:03.293890728 (94h45m3.164973783s) [upload=82602]: Got next segment. Inline: true Dec 9 14:14:03.293897561 (94h45m3.164980616s) [upload=82602]: Waiting for error group managing segments... Dec 9 14:14:03.294051774 (94h45m3.165134809s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:14:03.294059880 (94h45m3.165142925s) [upload=82602]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 14:14:03.294063066 (94h45m3.165146111s) [upload=82602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 14:15:01.261001342 (94h46m1.132084387s) [upload=82602]: Done uploading segments... Dec 9 14:15:01.261048982 (94h46m1.132132017s) [upload=82602]: Upload finished. err: metaclient: context deadline exceeded Duration:* 24.410311128s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)