Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 33.637760619s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:52:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 33.637788522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.XX..X...X....XXX....X....X.XX..X....XX.XX...............] duration: 6.148225675s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:52:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 6.148252475s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..X.XX..X...X....XXX....X....X.XX..X....XX.XX..............] duration: 25.708510043s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:52:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 25.708539448s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX] duration: 5.00111886s trace: https://jaeger.dev.storj.io/trace/a570af00b6fbf40 FAIL (2025-12-30 07:51:55 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001136714s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 18.007248103s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:51:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 18.00726768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..X.XX..X...X....XXX....X....X.XX..X....XX.XX.............] duration: 5.667796771s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:51:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 5.667828371s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 44.000746158s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:51:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 44.000771436s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [.X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.X] duration: 5.000962772s trace: https://jaeger.dev.storj.io/trace/6f554937c8a51d2c FAIL (2025-12-30 07:51:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0010666s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [..X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.] duration: 3.615956474s trace: https://jaeger.dev.storj.io/trace/7e75adec67fafb7f FAIL (2025-12-30 07:50:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.615982894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X..X.XX..X...X....XXX....X....X.XX..X....XX.XX............] duration: 5.668237702s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:50:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 5.668273118s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 21.91329559s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:50:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 21.913312573s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [...X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX] duration: 5.001665622s trace: https://jaeger.dev.storj.io/trace/7e75adec67fafb7f FAIL (2025-12-30 07:49:53 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001683906s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 50.72595334s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:49:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 50.725976453s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [....X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXX] duration: 5.002001361s trace: https://jaeger.dev.storj.io/trace/338caacdfafc8965 FAIL (2025-12-30 07:49:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002013584s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X..X.XX..X...X....XXX....X....X.XX..X....XX.XX...........] duration: 7.404994698s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:49:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 7.405027401s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 50.422096433s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:48:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 50.422125748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X...X..X.XX..X...X....XXX....X....X.XX..X....XX.XX..........] duration: 5.629074937s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:48:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 5.629101929s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [.....X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXX] duration: 5.001973681s trace: https://jaeger.dev.storj.io/trace/69f3438ddc840f09 FAIL (2025-12-30 07:48:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002000581s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X...X..X.XX..X...X....XXX....X....X.XX..X....XX.XX.........] duration: 5.633262082s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:48:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 5.633291077s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 41.780587879s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:48:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 41.780617475s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [X.....X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XXX] duration: 5.001253239s trace: https://jaeger.dev.storj.io/trace/216cbd8709aea2ce FAIL (2025-12-30 07:47:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001275361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X...X..X.XX..X...X....XXX....X....X.XX..X....XX.XX........] duration: 5.955247783s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:47:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 5.955274564s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 29.820418271s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:47:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 29.820450163s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [.X.....X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.XX] duration: 5.001841195s trace: https://jaeger.dev.storj.io/trace/1478126425f0ef3a FAIL (2025-12-30 07:47:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001868177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 31.364042737s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:46:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 31.364074498s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X.X.....X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.X] duration: 5.00088813s trace: https://jaeger.dev.storj.io/trace/273245664667d41d FAIL (2025-12-30 07:46:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00090973s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X...X..X.XX..X...X....XXX....X....X.XX..X....XX.XX.......] duration: 5.657493828s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:46:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 5.657527622s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.X.X.....X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X.] duration: 3.607793268s trace: https://jaeger.dev.storj.io/trace/26e743a966e0ab1f FAIL (2025-12-30 07:45:54 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.607822385s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 35.368400212s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:45:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 35.368417183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....X...X..X.XX..X...X....XXX....X....X.XX..X....XX.XX......] duration: 5.973577907s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:45:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 5.973614335s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 30.414495135s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:44:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 30.414511437s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..X.X.....X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.X] duration: 5.000927909s trace: https://jaeger.dev.storj.io/trace/26e743a966e0ab1f FAIL (2025-12-30 07:44:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000953086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 32.477369396s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:44:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 32.477399744s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....X...X..X.XX..X...X....XXX....X....X.XX..X....XX.XX.....] duration: 5.72597983s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:44:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 5.726009114s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 38.905705253s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:43:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 38.905733555s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [...X.X.....X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX.] duration: 3.36315525s trace: https://jaeger.dev.storj.io/trace/4306571174ddf4ae FAIL (2025-12-30 07:43:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.36321879s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.....X...X..X.XX..X...X....XXX....X....X.XX..X....XX.XX....] duration: 5.71523954s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:43:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 5.715265288s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.....X...X..X.XX..X...X....XXX....X....X.XX..X....XX.XX...] duration: 5.900469449s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:42:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 5.900488273s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 29.959566709s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:42:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 29.959598519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [....X.X.....X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XXX] duration: 5.001101839s trace: https://jaeger.dev.storj.io/trace/4306571174ddf4ae FAIL (2025-12-30 07:42:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001124401s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X.....X...X..X.XX..X...X....XXX....X....X.XX..X....XX.XX..] duration: 5.777933039s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:42:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 5.777960221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 27.759322164s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:42:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 27.759339978s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.....X.X.....X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.XX] duration: 5.000451398s trace: https://jaeger.dev.storj.io/trace/56cb526a13c89559 FAIL (2025-12-30 07:42:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000478291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..X.....X...X..X.XX..X...X....XXX....X....X.XX..X....XX.XX.] duration: 6.083328342s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:41:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 6.083347407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX..X.....X...X..X.XX..X...X....XXX....X....X.XX..X....XX.XX] duration: 30.004518968s trace: https://jaeger.dev.storj.io/trace/6d4d5a60e6f94955 FAIL (2025-12-30 07:41:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA) context deadline exceeded ======LOGS====== :40:49.054788681 (592h11m48.925871726s) [upload=519154]: Starting upload Dec 30 07:40:49.054885837 (592h11m48.925968872s) [upload=519154]: Done starting upload Dec 30 07:40:49.054934950 (592h11m48.926017995s) [upload=519154]: Uploading segments... Dec 30 07:40:49.055182181 (592h11m48.926265226s) [upload=519154]: Got next segment. Inline: false Dec 30 07:40:49.055223991 (592h11m48.926307036s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:49.055229451 (592h11m48.926312486s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:49.298135753 (592h11m49.169218798s) [seg_pos={0 0}, upload=519154]: Begin upload segment... Dec 30 07:40:49.298196328 (592h11m49.169279373s) [seg_pos={0 0}, upload=519154]: Done begin upload segment. Dec 30 07:40:49.298261663 (592h11m49.169344698s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.298276561 (592h11m49.169359636s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299156858 (592h11m49.170239903s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.299878671 (592h11m49.170961716s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.300572132 (592h11m49.171655177s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301104917 (592h11m49.172187962s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.301973872 (592h11m49.173056917s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.303384517 (592h11m49.174467552s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.304607015 (592h11m49.17569006s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.306120888 (592h11m49.177203933s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.308654973 (592h11m49.179738008s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.309907378 (592h11m49.180990423s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.311474632 (592h11m49.182557677s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.312790678 (592h11m49.183873713s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.313636559 (592h11m49.184719604s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:49.315395639 (592h11m49.186478684s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Uploading piece... Dec 30 07:40:51.015039829 (592h11m50.886122874s) [upload=519154]: Next returned nil segment Dec 30 07:40:51.015056301 (592h11m50.886139346s) [upload=519154]: Waiting for error group managing segments... Dec 30 07:40:51.295541269 (592h11m51.166624304s) [noise=true , address=145.79.227.10:4401 , piece_id=2MUANZZAA7RPGX3C2L3GC6GDBUIMDOYRMABSPUHJ3K7VJVSMIQBA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.306295497 (592h11m51.177378542s) [noise=true , address=145.79.227.16:4909 , piece_id=JGD3HFD63NJX2UJJ3EZTQENZOQCQ7WS2FWYPI2PFW3HMIRVNDK2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.309719476 (592h11m51.180802531s) [noise=true , address=145.79.227.29:4942 , piece_id=JSOITHSHKHUT5H2N2CYLCX2KTEILNFBYX4QTYRY3RD7BCDWSD7HQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.311558318 (592h11m51.182641363s) [noise=true , address=145.79.227.17:4971 , piece_id=DODDCLOEDXLVLBUR4DMMVO3NMNWUMK6U77ZO4AVAEYYMURC47XOQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315895245 (592h11m51.18697828s) [noise=true , address=145.79.227.11:4730 , piece_id=Q2VN5KIDPHNJE6MONXJFIJQYD345OW34AKYSHHFBU4K3REQYRP2Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.315913240 (592h11m51.186996305s) [noise=true , address=145.79.227.14:4839 , piece_id=23UIBH7DUV46VIBOM6G566MZ2KOY7ILMRV444QGQXVYRBTSK56FQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324171044 (592h11m51.195254089s) [noise=true , address=145.79.227.20:4439 , piece_id=YFVFFV7QGOHN7UL25YH43CJLK2O36LGPJMPBCT2UMNXVUIKKUZ3Q, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.324750969 (592h11m51.195834014s) [noise=true , address=145.79.227.4:4187 , piece_id=H5O4HB2REQEADCZPEB23NLPMRRBTE5M5QUKCHHMQ3IL6EV7U2AAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.373902841 (592h11m51.244985887s) [noise=true , address=145.79.227.37:4567 , piece_id=56UCQZRL2HIVMDSJFHQWI2YFSMQ3J33ZRNZ4QAFFKBPLNVKC3GJQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.382305040 (592h11m51.253388075s) [noise=true , address=145.79.227.3:4149 , piece_id=GP7SPJ72EZ22QMCSF6FC3PJI4MS46LIMTVUSPEJOZSJY3XAKB6JQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.399537454 (592h11m51.270620509s) [noise=true , address=145.79.227.47:4267 , piece_id=N4SHJQXKLRVSP4HDLIP52KCUWEXFXKU2SGJPCHMVKFKW4MN3DKAQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.616423298 (592h11m51.487506334s) [noise=true , address=45.91.116.7:4010 , piece_id=YRE67PIXX5LWZX5SJTQIG3MDGCL6PKOTA2JWFRX6XKNVKWCDSXGA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856656801 (592h11m51.727739846s) [noise=true , address=77.90.31.13:4629 , piece_id=XAEMLU3LDZTDOOR6V3WCAWHU5BBQX63GFTSCFVU2KE7MR67EJXHQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: <nil> Dec 30 07:40:51.856677201 (592h11m51.727760236s) [seg_pos={0 0}, upload=519154]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:51.856915535 (592h11m51.72799857s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856922798 (592h11m51.728005833s) [noise=true , address=145.79.227.34:4907 , piece_id=SNIUT7OHAGR7D7YA6R73D7ZDCEC66OU54KWRAPHZ7UZNTIHCU2KA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.856928439 (592h11m51.728011484s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:51.856938848 (592h11m51.728021893s) [noise=true , address=145.79.227.33:4870 , piece_id=KBLVTEQTPMYRZSR66CHNQQBHZRHASEKL4RH7XVJWXKVA4FKM4JSQ, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857140933 (592h11m51.728223978s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33478->45.91.116.10:4083: use of closed network connection Dec 30 07:40:51.857150251 (592h11m51.728233296s) [noise=true , address=45.91.116.10:4083 , piece_id=RYYYCDJM2FFYAJ26K4EO3KCIPC5FGJHATXEDPKONLZBVRNRYX5WA, seg_pos={0 0}, upload=519154]: Long tail cancellation due to optimal threshold Dec 30 07:40:51.857190938 (592h11m51.728273983s) [seg_pos={0 0}, upload=519154]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:51.857283094 (592h11m51.728366139s) [upload=519154]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:51.857289787 (592h11m51.728372832s) [upload=519154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:52.140725283 (592h11m52.011808329s) [upload=519154]: Done uploading segments... Dec 30 07:40:52.140743428 (592h11m52.011826473s) [upload=519154]: Upload finished. err: <nil> Duration: 30.004539306s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [......X.X.....X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.X] duration: 5.00145839s trace: https://jaeger.dev.storj.io/trace/65b7155c7d6adc69 FAIL (2025-12-30 07:41:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001487445s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 15.427200758s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:41:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 15.427220015s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 46.213842944s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:40:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 46.213860878s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.......X.X.....X..XXXXXX.X.....X........X...X...X.XX.XX.XXX.] duration: 4.342693304s trace: https://jaeger.dev.storj.io/trace/6d6dbdec1bb692b9 FAIL (2025-12-30 07:40:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.342743469s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXX..X.....X...X..X.XX..X...X....XXX....X....X.XX..X....XX.X] duration: 30.003972381s trace: https://jaeger.dev.storj.io/trace/6441f89fbf21502d FAIL (2025-12-30 07:40:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 62367 (592h11m15.633945412s) [upload=519147]: Done starting upload Dec 30 07:40:15.762898445 (592h11m15.633981481s) [upload=519147]: Uploading segments... Dec 30 07:40:15.763216572 (592h11m15.634299617s) [upload=519147]: Got next segment. Inline: false Dec 30 07:40:15.763289130 (592h11m15.634372175s) [upload=519147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:40:15.763296744 (592h11m15.63437979s) [upload=519147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:40:16.006796286 (592h11m15.877879331s) [seg_pos={0 0}, upload=519147]: Begin upload segment... Dec 30 07:40:16.006851050 (592h11m15.877934095s) [seg_pos={0 0}, upload=519147]: Done begin upload segment. Dec 30 07:40:16.006919831 (592h11m15.878002876s) [noise=true , address=145.79.227.15:4873 , piece_id=5EDWBFYYFVG4KWQE4XNGBL62VO7YSYLO72ITVPKDR4ME2QSKY7PA, seg_pos={0 0}, upload=519147]: Uploading piece... Dec 30 07:40:16.007748098 (592h11m15.878831143s) [noise=true , address=145.79.227.26:4654 , piece_id=BFGUNU272WKLBTGQBUDT5T33U3JY2UWABLVDPOKMB4TLNBWPKDXA, seg_pos={0 0}, upload=519147]: Uploading piece... Dec 30 07:40:16.008853813 (592h11m15.879936859s) [noise=true , address=77.90.31.11:4555 , piece_id=VQZ63YQ47X7CGIW2MC2VFTLGFMXUZSSBW5YC5VEWAN43ZNERR37A, seg_pos={0 0}, upload=519147]: Uploading piece... Dec 30 07:40:16.009301196 (592h11m15.880384241s) [noise=true , address=145.79.227.7:4293 , piece_id=2NZE7RROQUEQLXXGBPNPSB5UUBCVN44PIWNJRL2SRDIPWIJL7SYA, seg_pos={0 0}, upload=519147]: Uploading piece... Dec 30 07:40:16.009718721 (592h11m15.880801776s) [noise=true , address=145.79.227.11:4732 , piece_id=23EEWI64DCSBU3MFN4CS2UUXN362LPME5GBTC7F2BNRPBLTSH7MQ, seg_pos={0 0}, upload=519147]: Uploading piece... Dec 30 07:40:16.010015686 (592h11m15.881098731s) [noise=true , address=77.90.31.16:4735 , piece_id=XZJUOWXYQZ3BF7UDTBOKCX2SE3KUYMBSMO64LPNB5EBRNPN2JR4A, seg_pos={0 0}, upload=519147]: Uploading piece... Dec 30 07:40:16.010236477 (592h11m15.881319522s) [noise=true , address=45.91.116.10:4084 , piece_id=2BH64RIYE4ABOG26FO7M7KGFFJ6UVVFHV3ADXDQQD2YEUZ533QRA, seg_pos={0 0}, upload=519147]: Uploading piece... Dec 30 07:40:16.010692605 (592h11m15.88177564s) [noise=true , address=145.79.227.19:4079 , piece_id=26DX4GJ5UW5SUW4M6UKVVF7KWGSAOF3VUT7YHOBLB6BGZPC6TTQQ, seg_pos={0 0}, upload=519147]: Uploading piece... Dec 30 07:40:16.010729656 (592h11m15.881812701s) [noise=true , address=145.79.227.6:4257 , piece_id=2SMMQCYEVCQNWTRMLNFYEQVUK3C3YWZVZDV2TFYUEXGF2B3AAODA, seg_pos={0 0}, upload=519147]: Uploading piece... Dec 30 07:40:16.011061156 (592h11m15.882144201s) [noise=true , address=45.91.116.11:4120 , piece_id=343IGSWSF64SIWMVBLXCYSJFT45P5FPNL4ADYNGYOJXTHS5NGW6A, seg_pos={0 0}, upload=519147]: Uploading piece... Dec 30 07:40:16.011221272 (592h11m15.882304317s) [noise=true , address=145.79.227.42:4123 , piece_id=OB7R3C5KARN5RP737UOOBRMTBIROJF5AOVWIGZCOWHEN5SMSK6FA, seg_pos={0 0}, upload=519147]: Uploading piece... Dec 30 07:40:16.011390324 (592h11m15.882473359s) [noise=true , address=77.90.31.18:4805 , piece_id=LAN6G7S77Q5ECYHBPCIRBNL7YD7CQY47VXYGXF6RFIRSDUVFYRFQ, seg_pos={0 0}, upload=519147]: Uploading piece... Dec 30 07:40:16.011720812 (592h11m15.882803857s) [noise=true , address=145.79.227.34:4907 , piece_id=GGO4IVRBTY2L2XJW5RQ3WR65HHQVCCA4PGYV5EMYMX2RBCORL2LA, seg_pos={0 0}, upload=519147]: Uploading piece... Dec 30 07:40:16.011743386 (592h11m15.882826431s) [noise=true , address=145.79.227.20:4438 , piece_id=R6RNQMHNRCVGVEC2TJDLCWKGP7YFIE4AUNCGXVCD5MD5MR5655UA, seg_pos={0 0}, upload=519147]: Uploading piece... Dec 30 07:40:16.012057143 (592h11m15.883140178s) [noise=true , address=145.79.227.52:4446 , piece_id=2PKVEW326J4BT5M726LLILWOYEFZNUKQTB75NCEQKTG3LALVTREQ, seg_pos={0 0}, upload=519147]: Uploading piece... Dec 30 07:40:16.012369939 (592h11m15.883452984s) [noise=true , address=145.79.227.4:4185 , piece_id=4BHDFTORM6N2SYPTLQ7EXWAK4QN4X7HRH4EAAGBQAQERJCXS4P4Q, seg_pos={0 0}, upload=519147]: Uploading piece... Dec 30 07:40:17.687295673 (592h11m17.558378708s) [upload=519147]: Next returned nil segment Dec 30 07:40:17.687312465 (592h11m17.55839551s) [upload=519147]: Waiting for error group managing segments... Dec 30 07:40:17.987315879 (592h11m17.858398914s) [noise=true , address=145.79.227.19:4079 , piece_id=26DX4GJ5UW5SUW4M6UKVVF7KWGSAOF3VUT7YHOBLB6BGZPC6TTQQ, seg_pos={0 0}, upload=519147]: Done uploading piece. err: <nil> Dec 30 07:40:17.997931814 (592h11m17.869014859s) [noise=true , address=145.79.227.15:4873 , piece_id=5EDWBFYYFVG4KWQE4XNGBL62VO7YSYLO72ITVPKDR4ME2QSKY7PA, seg_pos={0 0}, upload=519147]: Done uploading piece. err: <nil> Dec 30 07:40:18.000432265 (592h11m17.87151532s) [noise=true , address=145.79.227.7:4293 , piece_id=2NZE7RROQUEQLXXGBPNPSB5UUBCVN44PIWNJRL2SRDIPWIJL7SYA, seg_pos={0 0}, upload=519147]: Done uploading piece. err: <nil> Dec 30 07:40:18.001285841 (592h11m17.872368896s) [noise=true , address=145.79.227.4:4185 , piece_id=4BHDFTORM6N2SYPTLQ7EXWAK4QN4X7HRH4EAAGBQAQERJCXS4P4Q, seg_pos={0 0}, upload=519147]: Done uploading piece. err: <nil> Dec 30 07:40:18.002095903 (592h11m17.873178958s) [noise=true , address=145.79.227.20:4438 , piece_id=R6RNQMHNRCVGVEC2TJDLCWKGP7YFIE4AUNCGXVCD5MD5MR5655UA, seg_pos={0 0}, upload=519147]: Done uploading piece. err: <nil> Dec 30 07:40:18.016318445 (592h11m17.8874015s) [noise=true , address=145.79.227.42:4123 , piece_id=OB7R3C5KARN5RP737UOOBRMTBIROJF5AOVWIGZCOWHEN5SMSK6FA, seg_pos={0 0}, upload=519147]: Done uploading piece. err: <nil> Dec 30 07:40:18.022962777 (592h11m17.894045822s) [noise=true , address=145.79.227.6:4257 , piece_id=2SMMQCYEVCQNWTRMLNFYEQVUK3C3YWZVZDV2TFYUEXGF2B3AAODA, seg_pos={0 0}, upload=519147]: Done uploading piece. err: <nil> Dec 30 07:40:18.023724618 (592h11m17.894807703s) [noise=true , address=145.79.227.11:4732 , piece_id=23EEWI64DCSBU3MFN4CS2UUXN362LPME5GBTC7F2BNRPBLTSH7MQ, seg_pos={0 0}, upload=519147]: Done uploading piece. err: <nil> Dec 30 07:40:18.025337819 (592h11m17.896420904s) [noise=true , address=145.79.227.52:4446 , piece_id=2PKVEW326J4BT5M726LLILWOYEFZNUKQTB75NCEQKTG3LALVTREQ, seg_pos={0 0}, upload=519147]: Done uploading piece. err: <nil> Dec 30 07:40:18.093458201 (592h11m17.964541246s) [noise=true , address=145.79.227.26:4654 , piece_id=BFGUNU272WKLBTGQBUDT5T33U3JY2UWABLVDPOKMB4TLNBWPKDXA, seg_pos={0 0}, upload=519147]: Done uploading piece. err: <nil> Dec 30 07:40:18.288789082 (592h11m18.159872127s) [noise=true , address=77.90.31.16:4735 , piece_id=XZJUOWXYQZ3BF7UDTBOKCX2SE3KUYMBSMO64LPNB5EBRNPN2JR4A, seg_pos={0 0}, upload=519147]: Done uploading piece. err: <nil> Dec 30 07:40:18.314826041 (592h11m18.185909076s) [noise=true , address=45.91.116.10:4084 , piece_id=2BH64RIYE4ABOG26FO7M7KGFFJ6UVVFHV3ADXDQQD2YEUZ533QRA, seg_pos={0 0}, upload=519147]: Done uploading piece. err: <nil> Dec 30 07:40:18.328625707 (592h11m18.199708742s) [noise=true , address=45.91.116.11:4120 , piece_id=343IGSWSF64SIWMVBLXCYSJFT45P5FPNL4ADYNGYOJXTHS5NGW6A, seg_pos={0 0}, upload=519147]: Done uploading piece. err: <nil> Dec 30 07:40:18.328645165 (592h11m18.1997282s) [seg_pos={0 0}, upload=519147]: Segment reached optimal threshold of 13 pieces. Dec 30 07:40:18.328934325 (592h11m18.20001737s) [noise=true , address=77.90.31.11:4555 , piece_id=VQZ63YQ47X7CGIW2MC2VFTLGFMXUZSSBW5YC5VEWAN43ZNERR37A, seg_pos={0 0}, upload=519147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: read tcp 188.166.225.170:34312->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 07:40:18.328941559 (592h11m18.200024594s) [noise=true , address=77.90.31.11:4555 , piece_id=VQZ63YQ47X7CGIW2MC2VFTLGFMXUZSSBW5YC5VEWAN43ZNERR37A, seg_pos={0 0}, upload=519147]: Long tail cancellation due to optimal threshold Dec 30 07:40:18.329184461 (592h11m18.200267496s) [noise=true , address=77.90.31.18:4805 , piece_id=LAN6G7S77Q5ECYHBPCIRBNL7YD7CQY47VXYGXF6RFIRSDUVFYRFQ, seg_pos={0 0}, upload=519147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:53700->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:53700->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53700->77.90.31.18:4805: use of closed network connection Dec 30 07:40:18.329190473 (592h11m18.200273518s) [noise=true , address=77.90.31.18:4805 , piece_id=LAN6G7S77Q5ECYHBPCIRBNL7YD7CQY47VXYGXF6RFIRSDUVFYRFQ, seg_pos={0 0}, upload=519147]: Long tail cancellation due to optimal threshold Dec 30 07:40:18.329297146 (592h11m18.200380181s) [noise=true , address=145.79.227.34:4907 , piece_id=GGO4IVRBTY2L2XJW5RQ3WR65HHQVCCA4PGYV5EMYMX2RBCORL2LA, seg_pos={0 0}, upload=519147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 07:40:18.329303298 (592h11m18.200386333s) [noise=true , address=145.79.227.34:4907 , piece_id=GGO4IVRBTY2L2XJW5RQ3WR65HHQVCCA4PGYV5EMYMX2RBCORL2LA, seg_pos={0 0}, upload=519147]: Long tail cancellation due to optimal threshold Dec 30 07:40:18.329345107 (592h11m18.200428152s) [seg_pos={0 0}, upload=519147]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:40:18.329417996 (592h11m18.200501031s) [upload=519147]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:40:18.329422905 (592h11m18.20050595s) [upload=519147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:40:18.618041200 (592h11m18.489124255s) [upload=519147]: Done uploading segments... Dec 30 07:40:18.618072280 (592h11m18.489155325s) [upload=519147]: Upload finished. err: <nil> Duration: 30.003994923s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)