Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................X...........X..X.....X] duration: 30.003556449s trace: https://jaeger.dev.storj.io/trace/31eb9423e47612bc FAIL (2025-12-18 08:38:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 08:37:40.460265422 (305h8m40.331348467s) [upload=267557]: Starting upload Dec 18 08:37:40.460340184 (305h8m40.331423229s) [upload=267557]: Done starting upload Dec 18 08:37:40.460533081 (305h8m40.331616116s) [upload=267557]: Uploading segments... Dec 18 08:37:40.460581564 (305h8m40.331664609s) [upload=267557]: Got next segment. Inline: false Dec 18 08:37:40.460608514 (305h8m40.331691549s) [upload=267557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:37:40.460612523 (305h8m40.331695558s) [upload=267557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:37:40.711682848 (305h8m40.582765883s) [seg_pos={0 0}, upload=267557]: Begin upload segment... Dec 18 08:37:40.711728534 (305h8m40.582811569s) [seg_pos={0 0}, upload=267557]: Done begin upload segment. Dec 18 08:37:40.711755816 (305h8m40.582838852s) [noise=true , address=77.90.31.14:4664 , piece_id=2SHSKSMOTULHN44HC3YGQRNPYZUZBBZLTYWJMSSQY2NVCJPWI3PA, seg_pos={0 0}, upload=267557]: Uploading piece... Dec 18 08:37:40.712510444 (305h8m40.583593509s) [noise=true , address=145.79.227.36:4519 , piece_id=JXZIAZ5MLY3UQDEY4BDO6B35WACE5BOSK7KBSFCUPEBZTQRI3Q3Q, seg_pos={0 0}, upload=267557]: Uploading piece... Dec 18 08:37:40.713478707 (305h8m40.584561752s) [noise=true , address=45.91.116.4:4940 , piece_id=24OP7WYLVHA7FROKQLYBNGW36E3U6I7HBYWXCNWHL3IBG3TFYXMQ, seg_pos={0 0}, upload=267557]: Uploading piece... Dec 18 08:37:40.714004839 (305h8m40.585087884s) [noise=true , address=77.90.31.16:4736 , piece_id=SD5CSZWVHXD5EMNMXLWARX446OSKBYLHFSEZQGGJZAPKDSTZ4G7Q, seg_pos={0 0}, upload=267557]: Uploading piece... Dec 18 08:37:40.714182788 (305h8m40.585265823s) [noise=true , address=145.79.227.29:4941 , piece_id=F3TAO3LYYM57NTT7IW43OC2EZG66SMZ4DCOVVNM6NOZUMYKLMB7A, seg_pos={0 0}, upload=267557]: Uploading piece... Dec 18 08:37:40.714471938 (305h8m40.585554973s) [noise=true , address=145.79.227.3:4151 , piece_id=N7AO7NTR33PQ4ZJD553626OYRD5Q7CDGIWPLXWOLIH3CCR2DMTOA, seg_pos={0 0}, upload=267557]: Uploading piece... Dec 18 08:37:40.714814962 (305h8m40.585897997s) [noise=true , address=145.79.227.38:4977 , piece_id=UGWSYG4WFYAFSYETGC5T6RWOYV34KGCEISXNTBDBZHTGTOCGEZGQ, seg_pos={0 0}, upload=267557]: Uploading piece... Dec 18 08:37:40.715171060 (305h8m40.586254095s) [noise=true , address=145.79.227.50:4373 , piece_id=ZHN6S2ZVDR6WLJ2VW4QXHHUYSZ4Y3FG3VTM77YFXUN2PBU2X5QOQ, seg_pos={0 0}, upload=267557]: Uploading piece... Dec 18 08:37:40.715220274 (305h8m40.586303309s) [noise=true , address=145.79.227.48:4301 , piece_id=K42SS3OJDYTYRLDADNAKRUPJIQVVIVT34KHFPEXXU7RBGSRVZQHA, seg_pos={0 0}, upload=267557]: Uploading piece... Dec 18 08:37:40.715501179 (305h8m40.586584214s) [noise=true , address=145.79.227.10:4401 , piece_id=P73CJ7FTJJATVIPWXUQQQBNQ5UTN76VAEFWEEOLA25ZSXBBTNO7Q, seg_pos={0 0}, upload=267557]: Uploading piece... Dec 18 08:37:40.715691881 (305h8m40.586774926s) [noise=true , address=45.91.116.9:4261 , piece_id=6E2ENJRULBY24AUASAKKK6CQ23DJ7G5HNYTI43HWJIXBZTU6UI6Q, seg_pos={0 0}, upload=267557]: Uploading piece... Dec 18 08:37:40.715953079 (305h8m40.587036114s) [noise=true , address=45.91.116.6:4974 , piece_id=EZNRT2VRQICUAE5KK2XD2BNLQU2TNY74A5HRDD3ZCDIE2S327A6Q, seg_pos={0 0}, upload=267557]: Uploading piece... Dec 18 08:37:40.716160324 (305h8m40.587243359s) [noise=true , address=45.91.116.11:4120 , piece_id=64BT3WCBNZ2BB3HMYDFIMYHTUKBTVKEQHVSVP75MWBIYETGCLXUA, seg_pos={0 0}, upload=267557]: Uploading piece... Dec 18 08:37:40.716635468 (305h8m40.587718513s) [noise=true , address=145.79.227.35:4481 , piece_id=6KPT5JONNPLWZVER3BZFHICZZGA4ISNEPR55V4GRIZLBYT6RVBDQ, seg_pos={0 0}, upload=267557]: Uploading piece... Dec 18 08:37:40.716713436 (305h8m40.587796481s) [noise=true , address=145.79.227.28:4725 , piece_id=OOXCKJQNTUFPIYCNLG4GIJYRGUUP6GYV3UFBPRDPID7KM7NTBRFQ, seg_pos={0 0}, upload=267557]: Uploading piece... Dec 18 08:37:40.717509131 (305h8m40.588592177s) [noise=true , address=145.79.227.16:4910 , piece_id=WMQ25KVHKLH7YLMOVVISQNC4G3EUSY34U5GVXTFXSUNPM4AOSPOQ, seg_pos={0 0}, upload=267557]: Uploading piece... Dec 18 08:37:42.414957522 (305h8m42.286040557s) [upload=267557]: Next returned nil segment Dec 18 08:37:42.415005303 (305h8m42.286088338s) [upload=267557]: Waiting for error group managing segments... Dec 18 08:37:42.693877036 (305h8m42.564960082s) [noise=true , address=145.79.227.36:4519 , piece_id=JXZIAZ5MLY3UQDEY4BDO6B35WACE5BOSK7KBSFCUPEBZTQRI3Q3Q, seg_pos={0 0}, upload=267557]: Done uploading piece. err: <nil> Dec 18 08:37:42.704469768 (305h8m42.575552813s) [noise=true , address=145.79.227.48:4301 , piece_id=K42SS3OJDYTYRLDADNAKRUPJIQVVIVT34KHFPEXXU7RBGSRVZQHA, seg_pos={0 0}, upload=267557]: Done uploading piece. err: <nil> Dec 18 08:37:42.704848719 (305h8m42.575931764s) [noise=true , address=145.79.227.35:4481 , piece_id=6KPT5JONNPLWZVER3BZFHICZZGA4ISNEPR55V4GRIZLBYT6RVBDQ, seg_pos={0 0}, upload=267557]: Done uploading piece. err: <nil> Dec 18 08:37:42.705120807 (305h8m42.576203852s) [noise=true , address=145.79.227.16:4910 , piece_id=WMQ25KVHKLH7YLMOVVISQNC4G3EUSY34U5GVXTFXSUNPM4AOSPOQ, seg_pos={0 0}, upload=267557]: Done uploading piece. err: <nil> Dec 18 08:37:42.719742419 (305h8m42.590825454s) [noise=true , address=145.79.227.50:4373 , piece_id=ZHN6S2ZVDR6WLJ2VW4QXHHUYSZ4Y3FG3VTM77YFXUN2PBU2X5QOQ, seg_pos={0 0}, upload=267557]: Done uploading piece. err: <nil> Dec 18 08:37:42.727213706 (305h8m42.598296751s) [noise=true , address=145.79.227.29:4941 , piece_id=F3TAO3LYYM57NTT7IW43OC2EZG66SMZ4DCOVVNM6NOZUMYKLMB7A, seg_pos={0 0}, upload=267557]: Done uploading piece. err: <nil> Dec 18 08:37:42.729047227 (305h8m42.600130272s) [noise=true , address=145.79.227.3:4151 , piece_id=N7AO7NTR33PQ4ZJD553626OYRD5Q7CDGIWPLXWOLIH3CCR2DMTOA, seg_pos={0 0}, upload=267557]: Done uploading piece. err: <nil> Dec 18 08:37:42.775562391 (305h8m42.646645436s) [noise=true , address=145.79.227.10:4401 , piece_id=P73CJ7FTJJATVIPWXUQQQBNQ5UTN76VAEFWEEOLA25ZSXBBTNO7Q, seg_pos={0 0}, upload=267557]: Done uploading piece. err: <nil> Dec 18 08:37:42.800893236 (305h8m42.671976271s) [noise=true , address=145.79.227.28:4725 , piece_id=OOXCKJQNTUFPIYCNLG4GIJYRGUUP6GYV3UFBPRDPID7KM7NTBRFQ, seg_pos={0 0}, upload=267557]: Done uploading piece. err: <nil> Dec 18 08:37:42.842078549 (305h8m42.713161584s) [noise=true , address=145.79.227.38:4977 , piece_id=UGWSYG4WFYAFSYETGC5T6RWOYV34KGCEISXNTBDBZHTGTOCGEZGQ, seg_pos={0 0}, upload=267557]: Done uploading piece. err: <nil> Dec 18 08:37:42.890175035 (305h8m42.76125808s) [noise=true , address=77.90.31.14:4664 , piece_id=2SHSKSMOTULHN44HC3YGQRNPYZUZBBZLTYWJMSSQY2NVCJPWI3PA, seg_pos={0 0}, upload=267557]: Done uploading piece. err: <nil> Dec 18 08:37:42.955382181 (305h8m42.826465216s) [noise=true , address=77.90.31.16:4736 , piece_id=SD5CSZWVHXD5EMNMXLWARX446OSKBYLHFSEZQGGJZAPKDSTZ4G7Q, seg_pos={0 0}, upload=267557]: Done uploading piece. err: <nil> Dec 18 08:37:42.958288846 (305h8m42.829371892s) [noise=true , address=45.91.116.9:4261 , piece_id=6E2ENJRULBY24AUASAKKK6CQ23DJ7G5HNYTI43HWJIXBZTU6UI6Q, seg_pos={0 0}, upload=267557]: Done uploading piece. err: <nil> Dec 18 08:37:42.958321930 (305h8m42.829404965s) [seg_pos={0 0}, upload=267557]: Segment reached optimal threshold of 13 pieces. Dec 18 08:37:42.958642209 (305h8m42.829725244s) [noise=true , address=45.91.116.4:4940 , piece_id=24OP7WYLVHA7FROKQLYBNGW36E3U6I7HBYWXCNWHL3IBG3TFYXMQ, seg_pos={0 0}, upload=267557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 18 08:37:42.958651146 (305h8m42.829734191s) [noise=true , address=45.91.116.4:4940 , piece_id=24OP7WYLVHA7FROKQLYBNGW36E3U6I7HBYWXCNWHL3IBG3TFYXMQ, seg_pos={0 0}, upload=267557]: Long tail cancellation due to optimal threshold Dec 18 08:37:42.958855425 (305h8m42.82993846s) [noise=true , address=45.91.116.6:4974 , piece_id=EZNRT2VRQICUAE5KK2XD2BNLQU2TNY74A5HRDD3ZCDIE2S327A6Q, seg_pos={0 0}, upload=267557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 18 08:37:42.958868770 (305h8m42.829951845s) [noise=true , address=45.91.116.6:4974 , piece_id=EZNRT2VRQICUAE5KK2XD2BNLQU2TNY74A5HRDD3ZCDIE2S327A6Q, seg_pos={0 0}, upload=267557]: Long tail cancellation due to optimal threshold Dec 18 08:37:42.958904238 (305h8m42.829987313s) [noise=true , address=45.91.116.11:4120 , piece_id=64BT3WCBNZ2BB3HMYDFIMYHTUKBTVKEQHVSVP75MWBIYETGCLXUA, seg_pos={0 0}, upload=267557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 18 08:37:42.958916672 (305h8m42.829999717s) [noise=true , address=45.91.116.11:4120 , piece_id=64BT3WCBNZ2BB3HMYDFIMYHTUKBTVKEQHVSVP75MWBIYETGCLXUA, seg_pos={0 0}, upload=267557]: Long tail cancellation due to optimal threshold Dec 18 08:37:42.958944064 (305h8m42.830027109s) [seg_pos={0 0}, upload=267557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:37:42.959031380 (305h8m42.830114415s) [upload=267557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:37:42.959037492 (305h8m42.830120527s) [upload=267557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:37:43.256007098 (305h8m43.127090143s) [upload=267557]: Done uploading segments... Dec 18 08:37:43.256030863 (305h8m43.127113908s) [upload=267557]: Upload finished. err: <nil> Duration: 30.003566667s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X...........X..X.....] duration: 26.048904771s trace: https://jaeger.dev.storj.io/trace/7e6640e8e766b46c FAIL (2025-12-18 08:37:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA) context deadline exceeded ======LOGS====== Dec 18 08:33:32.049014019 (305h4m31.920097054s) [upload=267496]: Starting upload Dec 18 08:33:32.049086377 (305h4m31.920169422s) [upload=267496]: Done starting upload Dec 18 08:33:32.049191306 (305h4m31.920274351s) [upload=267496]: Uploading segments... Dec 18 08:33:32.049419240 (305h4m31.920502285s) [upload=267496]: Got next segment. Inline: false Dec 18 08:33:32.049490996 (305h4m31.920574031s) [upload=267496]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:33:32.049497650 (305h4m31.920580685s) [upload=267496]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:33:32.295773565 (305h4m32.16685661s) [seg_pos={0 0}, upload=267496]: Begin upload segment... Dec 18 08:33:32.295846854 (305h4m32.166929889s) [seg_pos={0 0}, upload=267496]: Done begin upload segment. Dec 18 08:33:32.295885869 (305h4m32.166968904s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.295907280 (305h4m32.166990325s) [noise=true , address=145.79.227.41:4085 , piece_id=X5OEVD562CONOGZJUA56KXI6UBN4GNS24IY4XUXSMLJENM5H3DQA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.296542078 (305h4m32.167625113s) [noise=true , address=145.79.227.45:4231 , piece_id=TQHO636BRJK7JAFN2XIYPVTD47CQH4B5M342GLMCF7FBVAC2UH3Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.296984922 (305h4m32.168067967s) [noise=true , address=145.79.227.51:4411 , piece_id=EEHHLFQ6XNYKJQUT2F6Z43LA5YMHROA73WH74GV7YVT4AC5ZZUXQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.297424198 (305h4m32.168507233s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.298110775 (305h4m32.16919382s) [noise=true , address=77.90.31.13:4627 , piece_id=QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.298944673 (305h4m32.170027718s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSB6JVIP6OQK2WFJPSHXVMIDS7MFMUAPXH6KL6L4CF3UKBYSYQZA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.299922345 (305h4m32.17100538s) [noise=true , address=77.90.31.14:4663 , piece_id=LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.300515594 (305h4m32.171598639s) [noise=true , address=145.79.227.29:4941 , piece_id=OQZLSACE5VUVAUPR5W2TTMHPTLLDYJV3LUQ2REZNY7YUMRG2PQRQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.301300189 (305h4m32.172383234s) [noise=true , address=145.79.227.20:4439 , piece_id=FVN6LPYNPLZ6WNNH3MYQK3LHKXERKUUYDPZYPAX5U3PWZSTF6LQA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.302242293 (305h4m32.173325338s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.302644288 (305h4m32.173727333s) [noise=true , address=145.79.227.14:4839 , piece_id=I2MDVDOJY7JAHN3YWEKIDETZAOPSGJK4CCQNULXC5R3VDXIFKT5Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.303913255 (305h4m32.1749963s) [noise=true , address=145.79.227.34:4905 , piece_id=SXWZTRLOU62WIGIDVMXEYVNM3LVXJNJC4Z5CEOVKXZFVLPI7KD2A, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.305187481 (305h4m32.176270526s) [noise=true , address=145.79.227.39:4015 , piece_id=5YJ6TIMERXKG5XCOZUNLNJLRFPIKARX3EEZ7NDL6YIBPSUZZEYHQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.306724738 (305h4m32.177807783s) [noise=true , address=145.79.227.31:4798 , piece_id=V4QMNVKADDFQPVT6FZBODVS5ZCYK524OWUVRDBU76GJNBDVA24CA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.309176456 (305h4m32.180259501s) [noise=true , address=145.79.227.36:4517 , piece_id=NKXOC3YTQA22MMKQSYVR7IJOOWPZSBJV3C3WW6ZUV3B7BFIWJJUA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:34.010761764 (305h4m33.88184481s) [upload=267496]: Next returned nil segment Dec 18 08:33:34.010780831 (305h4m33.881863896s) [upload=267496]: Waiting for error group managing segments... Dec 18 08:33:34.291265406 (305h4m34.162348471s) [noise=true , address=145.79.227.45:4231 , piece_id=TQHO636BRJK7JAFN2XIYPVTD47CQH4B5M342GLMCF7FBVAC2UH3Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.293371326 (305h4m34.164454381s) [noise=true , address=145.79.227.41:4085 , piece_id=X5OEVD562CONOGZJUA56KXI6UBN4GNS24IY4XUXSMLJENM5H3DQA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.295285591 (305h4m34.166368637s) [noise=true , address=145.79.227.14:4839 , piece_id=I2MDVDOJY7JAHN3YWEKIDETZAOPSGJK4CCQNULXC5R3VDXIFKT5Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.295870796 (305h4m34.166953841s) [noise=true , address=145.79.227.31:4798 , piece_id=V4QMNVKADDFQPVT6FZBODVS5ZCYK524OWUVRDBU76GJNBDVA24CA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.307409507 (305h4m34.178492582s) [noise=true , address=145.79.227.51:4411 , piece_id=EEHHLFQ6XNYKJQUT2F6Z43LA5YMHROA73WH74GV7YVT4AC5ZZUXQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.310409920 (305h4m34.181492965s) [noise=true , address=145.79.227.39:4015 , piece_id=5YJ6TIMERXKG5XCOZUNLNJLRFPIKARX3EEZ7NDL6YIBPSUZZEYHQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.315729651 (305h4m34.186812696s) [noise=true , address=145.79.227.34:4905 , piece_id=SXWZTRLOU62WIGIDVMXEYVNM3LVXJNJC4Z5CEOVKXZFVLPI7KD2A, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.317510652 (305h4m34.188593697s) [noise=true , address=145.79.227.36:4517 , piece_id=NKXOC3YTQA22MMKQSYVR7IJOOWPZSBJV3C3WW6ZUV3B7BFIWJJUA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.320827428 (305h4m34.191910473s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSB6JVIP6OQK2WFJPSHXVMIDS7MFMUAPXH6KL6L4CF3UKBYSYQZA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.374645999 (305h4m34.245729044s) [noise=true , address=145.79.227.20:4439 , piece_id=FVN6LPYNPLZ6WNNH3MYQK3LHKXERKUUYDPZYPAX5U3PWZSTF6LQA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.387615185 (305h4m34.25869823s) [noise=true , address=145.79.227.29:4941 , piece_id=OQZLSACE5VUVAUPR5W2TTMHPTLLDYJV3LUQ2REZNY7YUMRG2PQRQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.499570748 (305h4m34.370653793s) [noise=true , address=77.90.31.14:4663 , piece_id=LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.586131707 (305h4m34.457214752s) [noise=true , address=77.90.31.13:4627 , piece_id=QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.586162034 (305h4m34.45724508s) [seg_pos={0 0}, upload=267496]: Segment reached optimal threshold of 13 pieces. Dec 18 08:33:34.586441837 (305h4m34.457524892s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586451295 (305h4m34.45753434s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586829616 (305h4m34.457912671s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586850696 (305h4m34.457933741s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586885893 (305h4m34.457968938s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586896503 (305h4m34.457979538s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586946888 (305h4m34.458029924s) [seg_pos={0 0}, upload=267496]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:33:34.587023585 (305h4m34.45810663s) [upload=267496]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:33:34.587029046 (305h4m34.458112081s) [upload=267496]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:33:34.882748803 (305h4m34.753831858s) [upload=267496]: Done uploading segments... Dec 18 08:33:34.882773580 (305h4m34.753856625s) [upload=267496]: Upload finished. err: <nil> Duration: 26.048934247s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X...........X..X....] duration: 5.891890874s trace: https://jaeger.dev.storj.io/trace/7e6640e8e766b46c FAIL (2025-12-18 08:36:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA) context deadline exceeded ======LOGS====== Dec 18 08:33:32.049014019 (305h4m31.920097054s) [upload=267496]: Starting upload Dec 18 08:33:32.049086377 (305h4m31.920169422s) [upload=267496]: Done starting upload Dec 18 08:33:32.049191306 (305h4m31.920274351s) [upload=267496]: Uploading segments... Dec 18 08:33:32.049419240 (305h4m31.920502285s) [upload=267496]: Got next segment. Inline: false Dec 18 08:33:32.049490996 (305h4m31.920574031s) [upload=267496]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:33:32.049497650 (305h4m31.920580685s) [upload=267496]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:33:32.295773565 (305h4m32.16685661s) [seg_pos={0 0}, upload=267496]: Begin upload segment... Dec 18 08:33:32.295846854 (305h4m32.166929889s) [seg_pos={0 0}, upload=267496]: Done begin upload segment. Dec 18 08:33:32.295885869 (305h4m32.166968904s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.295907280 (305h4m32.166990325s) [noise=true , address=145.79.227.41:4085 , piece_id=X5OEVD562CONOGZJUA56KXI6UBN4GNS24IY4XUXSMLJENM5H3DQA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.296542078 (305h4m32.167625113s) [noise=true , address=145.79.227.45:4231 , piece_id=TQHO636BRJK7JAFN2XIYPVTD47CQH4B5M342GLMCF7FBVAC2UH3Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.296984922 (305h4m32.168067967s) [noise=true , address=145.79.227.51:4411 , piece_id=EEHHLFQ6XNYKJQUT2F6Z43LA5YMHROA73WH74GV7YVT4AC5ZZUXQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.297424198 (305h4m32.168507233s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.298110775 (305h4m32.16919382s) [noise=true , address=77.90.31.13:4627 , piece_id=QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.298944673 (305h4m32.170027718s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSB6JVIP6OQK2WFJPSHXVMIDS7MFMUAPXH6KL6L4CF3UKBYSYQZA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.299922345 (305h4m32.17100538s) [noise=true , address=77.90.31.14:4663 , piece_id=LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.300515594 (305h4m32.171598639s) [noise=true , address=145.79.227.29:4941 , piece_id=OQZLSACE5VUVAUPR5W2TTMHPTLLDYJV3LUQ2REZNY7YUMRG2PQRQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.301300189 (305h4m32.172383234s) [noise=true , address=145.79.227.20:4439 , piece_id=FVN6LPYNPLZ6WNNH3MYQK3LHKXERKUUYDPZYPAX5U3PWZSTF6LQA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.302242293 (305h4m32.173325338s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.302644288 (305h4m32.173727333s) [noise=true , address=145.79.227.14:4839 , piece_id=I2MDVDOJY7JAHN3YWEKIDETZAOPSGJK4CCQNULXC5R3VDXIFKT5Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.303913255 (305h4m32.1749963s) [noise=true , address=145.79.227.34:4905 , piece_id=SXWZTRLOU62WIGIDVMXEYVNM3LVXJNJC4Z5CEOVKXZFVLPI7KD2A, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.305187481 (305h4m32.176270526s) [noise=true , address=145.79.227.39:4015 , piece_id=5YJ6TIMERXKG5XCOZUNLNJLRFPIKARX3EEZ7NDL6YIBPSUZZEYHQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.306724738 (305h4m32.177807783s) [noise=true , address=145.79.227.31:4798 , piece_id=V4QMNVKADDFQPVT6FZBODVS5ZCYK524OWUVRDBU76GJNBDVA24CA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.309176456 (305h4m32.180259501s) [noise=true , address=145.79.227.36:4517 , piece_id=NKXOC3YTQA22MMKQSYVR7IJOOWPZSBJV3C3WW6ZUV3B7BFIWJJUA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:34.010761764 (305h4m33.88184481s) [upload=267496]: Next returned nil segment Dec 18 08:33:34.010780831 (305h4m33.881863896s) [upload=267496]: Waiting for error group managing segments... Dec 18 08:33:34.291265406 (305h4m34.162348471s) [noise=true , address=145.79.227.45:4231 , piece_id=TQHO636BRJK7JAFN2XIYPVTD47CQH4B5M342GLMCF7FBVAC2UH3Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.293371326 (305h4m34.164454381s) [noise=true , address=145.79.227.41:4085 , piece_id=X5OEVD562CONOGZJUA56KXI6UBN4GNS24IY4XUXSMLJENM5H3DQA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.295285591 (305h4m34.166368637s) [noise=true , address=145.79.227.14:4839 , piece_id=I2MDVDOJY7JAHN3YWEKIDETZAOPSGJK4CCQNULXC5R3VDXIFKT5Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.295870796 (305h4m34.166953841s) [noise=true , address=145.79.227.31:4798 , piece_id=V4QMNVKADDFQPVT6FZBODVS5ZCYK524OWUVRDBU76GJNBDVA24CA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.307409507 (305h4m34.178492582s) [noise=true , address=145.79.227.51:4411 , piece_id=EEHHLFQ6XNYKJQUT2F6Z43LA5YMHROA73WH74GV7YVT4AC5ZZUXQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.310409920 (305h4m34.181492965s) [noise=true , address=145.79.227.39:4015 , piece_id=5YJ6TIMERXKG5XCOZUNLNJLRFPIKARX3EEZ7NDL6YIBPSUZZEYHQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.315729651 (305h4m34.186812696s) [noise=true , address=145.79.227.34:4905 , piece_id=SXWZTRLOU62WIGIDVMXEYVNM3LVXJNJC4Z5CEOVKXZFVLPI7KD2A, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.317510652 (305h4m34.188593697s) [noise=true , address=145.79.227.36:4517 , piece_id=NKXOC3YTQA22MMKQSYVR7IJOOWPZSBJV3C3WW6ZUV3B7BFIWJJUA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.320827428 (305h4m34.191910473s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSB6JVIP6OQK2WFJPSHXVMIDS7MFMUAPXH6KL6L4CF3UKBYSYQZA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.374645999 (305h4m34.245729044s) [noise=true , address=145.79.227.20:4439 , piece_id=FVN6LPYNPLZ6WNNH3MYQK3LHKXERKUUYDPZYPAX5U3PWZSTF6LQA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.387615185 (305h4m34.25869823s) [noise=true , address=145.79.227.29:4941 , piece_id=OQZLSACE5VUVAUPR5W2TTMHPTLLDYJV3LUQ2REZNY7YUMRG2PQRQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.499570748 (305h4m34.370653793s) [noise=true , address=77.90.31.14:4663 , piece_id=LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.586131707 (305h4m34.457214752s) [noise=true , address=77.90.31.13:4627 , piece_id=QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.586162034 (305h4m34.45724508s) [seg_pos={0 0}, upload=267496]: Segment reached optimal threshold of 13 pieces. Dec 18 08:33:34.586441837 (305h4m34.457524892s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586451295 (305h4m34.45753434s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586829616 (305h4m34.457912671s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586850696 (305h4m34.457933741s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586885893 (305h4m34.457968938s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586896503 (305h4m34.457979538s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586946888 (305h4m34.458029924s) [seg_pos={0 0}, upload=267496]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:33:34.587023585 (305h4m34.45810663s) [upload=267496]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:33:34.587029046 (305h4m34.458112081s) [upload=267496]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:33:34.882748803 (305h4m34.753831858s) [upload=267496]: Done uploading segments... Dec 18 08:33:34.882773580 (305h4m34.753856625s) [upload=267496]: Upload finished. err: <nil> Duration: 5.891921022s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X...........X..X...] duration: 5.803827998s trace: https://jaeger.dev.storj.io/trace/7e6640e8e766b46c FAIL (2025-12-18 08:35:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA) context deadline exceeded ======LOGS====== Dec 18 08:33:32.049014019 (305h4m31.920097054s) [upload=267496]: Starting upload Dec 18 08:33:32.049086377 (305h4m31.920169422s) [upload=267496]: Done starting upload Dec 18 08:33:32.049191306 (305h4m31.920274351s) [upload=267496]: Uploading segments... Dec 18 08:33:32.049419240 (305h4m31.920502285s) [upload=267496]: Got next segment. Inline: false Dec 18 08:33:32.049490996 (305h4m31.920574031s) [upload=267496]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:33:32.049497650 (305h4m31.920580685s) [upload=267496]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:33:32.295773565 (305h4m32.16685661s) [seg_pos={0 0}, upload=267496]: Begin upload segment... Dec 18 08:33:32.295846854 (305h4m32.166929889s) [seg_pos={0 0}, upload=267496]: Done begin upload segment. Dec 18 08:33:32.295885869 (305h4m32.166968904s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.295907280 (305h4m32.166990325s) [noise=true , address=145.79.227.41:4085 , piece_id=X5OEVD562CONOGZJUA56KXI6UBN4GNS24IY4XUXSMLJENM5H3DQA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.296542078 (305h4m32.167625113s) [noise=true , address=145.79.227.45:4231 , piece_id=TQHO636BRJK7JAFN2XIYPVTD47CQH4B5M342GLMCF7FBVAC2UH3Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.296984922 (305h4m32.168067967s) [noise=true , address=145.79.227.51:4411 , piece_id=EEHHLFQ6XNYKJQUT2F6Z43LA5YMHROA73WH74GV7YVT4AC5ZZUXQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.297424198 (305h4m32.168507233s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.298110775 (305h4m32.16919382s) [noise=true , address=77.90.31.13:4627 , piece_id=QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.298944673 (305h4m32.170027718s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSB6JVIP6OQK2WFJPSHXVMIDS7MFMUAPXH6KL6L4CF3UKBYSYQZA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.299922345 (305h4m32.17100538s) [noise=true , address=77.90.31.14:4663 , piece_id=LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.300515594 (305h4m32.171598639s) [noise=true , address=145.79.227.29:4941 , piece_id=OQZLSACE5VUVAUPR5W2TTMHPTLLDYJV3LUQ2REZNY7YUMRG2PQRQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.301300189 (305h4m32.172383234s) [noise=true , address=145.79.227.20:4439 , piece_id=FVN6LPYNPLZ6WNNH3MYQK3LHKXERKUUYDPZYPAX5U3PWZSTF6LQA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.302242293 (305h4m32.173325338s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.302644288 (305h4m32.173727333s) [noise=true , address=145.79.227.14:4839 , piece_id=I2MDVDOJY7JAHN3YWEKIDETZAOPSGJK4CCQNULXC5R3VDXIFKT5Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.303913255 (305h4m32.1749963s) [noise=true , address=145.79.227.34:4905 , piece_id=SXWZTRLOU62WIGIDVMXEYVNM3LVXJNJC4Z5CEOVKXZFVLPI7KD2A, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.305187481 (305h4m32.176270526s) [noise=true , address=145.79.227.39:4015 , piece_id=5YJ6TIMERXKG5XCOZUNLNJLRFPIKARX3EEZ7NDL6YIBPSUZZEYHQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.306724738 (305h4m32.177807783s) [noise=true , address=145.79.227.31:4798 , piece_id=V4QMNVKADDFQPVT6FZBODVS5ZCYK524OWUVRDBU76GJNBDVA24CA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.309176456 (305h4m32.180259501s) [noise=true , address=145.79.227.36:4517 , piece_id=NKXOC3YTQA22MMKQSYVR7IJOOWPZSBJV3C3WW6ZUV3B7BFIWJJUA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:34.010761764 (305h4m33.88184481s) [upload=267496]: Next returned nil segment Dec 18 08:33:34.010780831 (305h4m33.881863896s) [upload=267496]: Waiting for error group managing segments... Dec 18 08:33:34.291265406 (305h4m34.162348471s) [noise=true , address=145.79.227.45:4231 , piece_id=TQHO636BRJK7JAFN2XIYPVTD47CQH4B5M342GLMCF7FBVAC2UH3Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.293371326 (305h4m34.164454381s) [noise=true , address=145.79.227.41:4085 , piece_id=X5OEVD562CONOGZJUA56KXI6UBN4GNS24IY4XUXSMLJENM5H3DQA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.295285591 (305h4m34.166368637s) [noise=true , address=145.79.227.14:4839 , piece_id=I2MDVDOJY7JAHN3YWEKIDETZAOPSGJK4CCQNULXC5R3VDXIFKT5Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.295870796 (305h4m34.166953841s) [noise=true , address=145.79.227.31:4798 , piece_id=V4QMNVKADDFQPVT6FZBODVS5ZCYK524OWUVRDBU76GJNBDVA24CA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.307409507 (305h4m34.178492582s) [noise=true , address=145.79.227.51:4411 , piece_id=EEHHLFQ6XNYKJQUT2F6Z43LA5YMHROA73WH74GV7YVT4AC5ZZUXQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.310409920 (305h4m34.181492965s) [noise=true , address=145.79.227.39:4015 , piece_id=5YJ6TIMERXKG5XCOZUNLNJLRFPIKARX3EEZ7NDL6YIBPSUZZEYHQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.315729651 (305h4m34.186812696s) [noise=true , address=145.79.227.34:4905 , piece_id=SXWZTRLOU62WIGIDVMXEYVNM3LVXJNJC4Z5CEOVKXZFVLPI7KD2A, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.317510652 (305h4m34.188593697s) [noise=true , address=145.79.227.36:4517 , piece_id=NKXOC3YTQA22MMKQSYVR7IJOOWPZSBJV3C3WW6ZUV3B7BFIWJJUA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.320827428 (305h4m34.191910473s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSB6JVIP6OQK2WFJPSHXVMIDS7MFMUAPXH6KL6L4CF3UKBYSYQZA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.374645999 (305h4m34.245729044s) [noise=true , address=145.79.227.20:4439 , piece_id=FVN6LPYNPLZ6WNNH3MYQK3LHKXERKUUYDPZYPAX5U3PWZSTF6LQA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.387615185 (305h4m34.25869823s) [noise=true , address=145.79.227.29:4941 , piece_id=OQZLSACE5VUVAUPR5W2TTMHPTLLDYJV3LUQ2REZNY7YUMRG2PQRQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.499570748 (305h4m34.370653793s) [noise=true , address=77.90.31.14:4663 , piece_id=LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.586131707 (305h4m34.457214752s) [noise=true , address=77.90.31.13:4627 , piece_id=QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.586162034 (305h4m34.45724508s) [seg_pos={0 0}, upload=267496]: Segment reached optimal threshold of 13 pieces. Dec 18 08:33:34.586441837 (305h4m34.457524892s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586451295 (305h4m34.45753434s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586829616 (305h4m34.457912671s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586850696 (305h4m34.457933741s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586885893 (305h4m34.457968938s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586896503 (305h4m34.457979538s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586946888 (305h4m34.458029924s) [seg_pos={0 0}, upload=267496]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:33:34.587023585 (305h4m34.45810663s) [upload=267496]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:33:34.587029046 (305h4m34.458112081s) [upload=267496]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:33:34.882748803 (305h4m34.753831858s) [upload=267496]: Done uploading segments... Dec 18 08:33:34.882773580 (305h4m34.753856625s) [upload=267496]: Upload finished. err: <nil> Duration: 5.803850661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X...........X..X..] duration: 6.10862641s trace: https://jaeger.dev.storj.io/trace/7e6640e8e766b46c FAIL (2025-12-18 08:35:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA) context deadline exceeded ======LOGS====== Dec 18 08:33:32.049014019 (305h4m31.920097054s) [upload=267496]: Starting upload Dec 18 08:33:32.049086377 (305h4m31.920169422s) [upload=267496]: Done starting upload Dec 18 08:33:32.049191306 (305h4m31.920274351s) [upload=267496]: Uploading segments... Dec 18 08:33:32.049419240 (305h4m31.920502285s) [upload=267496]: Got next segment. Inline: false Dec 18 08:33:32.049490996 (305h4m31.920574031s) [upload=267496]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:33:32.049497650 (305h4m31.920580685s) [upload=267496]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:33:32.295773565 (305h4m32.16685661s) [seg_pos={0 0}, upload=267496]: Begin upload segment... Dec 18 08:33:32.295846854 (305h4m32.166929889s) [seg_pos={0 0}, upload=267496]: Done begin upload segment. Dec 18 08:33:32.295885869 (305h4m32.166968904s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.295907280 (305h4m32.166990325s) [noise=true , address=145.79.227.41:4085 , piece_id=X5OEVD562CONOGZJUA56KXI6UBN4GNS24IY4XUXSMLJENM5H3DQA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.296542078 (305h4m32.167625113s) [noise=true , address=145.79.227.45:4231 , piece_id=TQHO636BRJK7JAFN2XIYPVTD47CQH4B5M342GLMCF7FBVAC2UH3Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.296984922 (305h4m32.168067967s) [noise=true , address=145.79.227.51:4411 , piece_id=EEHHLFQ6XNYKJQUT2F6Z43LA5YMHROA73WH74GV7YVT4AC5ZZUXQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.297424198 (305h4m32.168507233s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.298110775 (305h4m32.16919382s) [noise=true , address=77.90.31.13:4627 , piece_id=QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.298944673 (305h4m32.170027718s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSB6JVIP6OQK2WFJPSHXVMIDS7MFMUAPXH6KL6L4CF3UKBYSYQZA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.299922345 (305h4m32.17100538s) [noise=true , address=77.90.31.14:4663 , piece_id=LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.300515594 (305h4m32.171598639s) [noise=true , address=145.79.227.29:4941 , piece_id=OQZLSACE5VUVAUPR5W2TTMHPTLLDYJV3LUQ2REZNY7YUMRG2PQRQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.301300189 (305h4m32.172383234s) [noise=true , address=145.79.227.20:4439 , piece_id=FVN6LPYNPLZ6WNNH3MYQK3LHKXERKUUYDPZYPAX5U3PWZSTF6LQA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.302242293 (305h4m32.173325338s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.302644288 (305h4m32.173727333s) [noise=true , address=145.79.227.14:4839 , piece_id=I2MDVDOJY7JAHN3YWEKIDETZAOPSGJK4CCQNULXC5R3VDXIFKT5Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.303913255 (305h4m32.1749963s) [noise=true , address=145.79.227.34:4905 , piece_id=SXWZTRLOU62WIGIDVMXEYVNM3LVXJNJC4Z5CEOVKXZFVLPI7KD2A, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.305187481 (305h4m32.176270526s) [noise=true , address=145.79.227.39:4015 , piece_id=5YJ6TIMERXKG5XCOZUNLNJLRFPIKARX3EEZ7NDL6YIBPSUZZEYHQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.306724738 (305h4m32.177807783s) [noise=true , address=145.79.227.31:4798 , piece_id=V4QMNVKADDFQPVT6FZBODVS5ZCYK524OWUVRDBU76GJNBDVA24CA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.309176456 (305h4m32.180259501s) [noise=true , address=145.79.227.36:4517 , piece_id=NKXOC3YTQA22MMKQSYVR7IJOOWPZSBJV3C3WW6ZUV3B7BFIWJJUA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:34.010761764 (305h4m33.88184481s) [upload=267496]: Next returned nil segment Dec 18 08:33:34.010780831 (305h4m33.881863896s) [upload=267496]: Waiting for error group managing segments... Dec 18 08:33:34.291265406 (305h4m34.162348471s) [noise=true , address=145.79.227.45:4231 , piece_id=TQHO636BRJK7JAFN2XIYPVTD47CQH4B5M342GLMCF7FBVAC2UH3Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.293371326 (305h4m34.164454381s) [noise=true , address=145.79.227.41:4085 , piece_id=X5OEVD562CONOGZJUA56KXI6UBN4GNS24IY4XUXSMLJENM5H3DQA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.295285591 (305h4m34.166368637s) [noise=true , address=145.79.227.14:4839 , piece_id=I2MDVDOJY7JAHN3YWEKIDETZAOPSGJK4CCQNULXC5R3VDXIFKT5Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.295870796 (305h4m34.166953841s) [noise=true , address=145.79.227.31:4798 , piece_id=V4QMNVKADDFQPVT6FZBODVS5ZCYK524OWUVRDBU76GJNBDVA24CA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.307409507 (305h4m34.178492582s) [noise=true , address=145.79.227.51:4411 , piece_id=EEHHLFQ6XNYKJQUT2F6Z43LA5YMHROA73WH74GV7YVT4AC5ZZUXQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.310409920 (305h4m34.181492965s) [noise=true , address=145.79.227.39:4015 , piece_id=5YJ6TIMERXKG5XCOZUNLNJLRFPIKARX3EEZ7NDL6YIBPSUZZEYHQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.315729651 (305h4m34.186812696s) [noise=true , address=145.79.227.34:4905 , piece_id=SXWZTRLOU62WIGIDVMXEYVNM3LVXJNJC4Z5CEOVKXZFVLPI7KD2A, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.317510652 (305h4m34.188593697s) [noise=true , address=145.79.227.36:4517 , piece_id=NKXOC3YTQA22MMKQSYVR7IJOOWPZSBJV3C3WW6ZUV3B7BFIWJJUA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.320827428 (305h4m34.191910473s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSB6JVIP6OQK2WFJPSHXVMIDS7MFMUAPXH6KL6L4CF3UKBYSYQZA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.374645999 (305h4m34.245729044s) [noise=true , address=145.79.227.20:4439 , piece_id=FVN6LPYNPLZ6WNNH3MYQK3LHKXERKUUYDPZYPAX5U3PWZSTF6LQA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.387615185 (305h4m34.25869823s) [noise=true , address=145.79.227.29:4941 , piece_id=OQZLSACE5VUVAUPR5W2TTMHPTLLDYJV3LUQ2REZNY7YUMRG2PQRQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.499570748 (305h4m34.370653793s) [noise=true , address=77.90.31.14:4663 , piece_id=LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.586131707 (305h4m34.457214752s) [noise=true , address=77.90.31.13:4627 , piece_id=QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.586162034 (305h4m34.45724508s) [seg_pos={0 0}, upload=267496]: Segment reached optimal threshold of 13 pieces. Dec 18 08:33:34.586441837 (305h4m34.457524892s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586451295 (305h4m34.45753434s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586829616 (305h4m34.457912671s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586850696 (305h4m34.457933741s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586885893 (305h4m34.457968938s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586896503 (305h4m34.457979538s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586946888 (305h4m34.458029924s) [seg_pos={0 0}, upload=267496]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:33:34.587023585 (305h4m34.45810663s) [upload=267496]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:33:34.587029046 (305h4m34.458112081s) [upload=267496]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:33:34.882748803 (305h4m34.753831858s) [upload=267496]: Done uploading segments... Dec 18 08:33:34.882773580 (305h4m34.753856625s) [upload=267496]: Upload finished. err: <nil> Duration: 6.108655626s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X...........X..X.] duration: 15.815066192s trace: https://jaeger.dev.storj.io/trace/7e6640e8e766b46c FAIL (2025-12-18 08:34:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA) context deadline exceeded ======LOGS====== Dec 18 08:33:32.049014019 (305h4m31.920097054s) [upload=267496]: Starting upload Dec 18 08:33:32.049086377 (305h4m31.920169422s) [upload=267496]: Done starting upload Dec 18 08:33:32.049191306 (305h4m31.920274351s) [upload=267496]: Uploading segments... Dec 18 08:33:32.049419240 (305h4m31.920502285s) [upload=267496]: Got next segment. Inline: false Dec 18 08:33:32.049490996 (305h4m31.920574031s) [upload=267496]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:33:32.049497650 (305h4m31.920580685s) [upload=267496]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:33:32.295773565 (305h4m32.16685661s) [seg_pos={0 0}, upload=267496]: Begin upload segment... Dec 18 08:33:32.295846854 (305h4m32.166929889s) [seg_pos={0 0}, upload=267496]: Done begin upload segment. Dec 18 08:33:32.295885869 (305h4m32.166968904s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.295907280 (305h4m32.166990325s) [noise=true , address=145.79.227.41:4085 , piece_id=X5OEVD562CONOGZJUA56KXI6UBN4GNS24IY4XUXSMLJENM5H3DQA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.296542078 (305h4m32.167625113s) [noise=true , address=145.79.227.45:4231 , piece_id=TQHO636BRJK7JAFN2XIYPVTD47CQH4B5M342GLMCF7FBVAC2UH3Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.296984922 (305h4m32.168067967s) [noise=true , address=145.79.227.51:4411 , piece_id=EEHHLFQ6XNYKJQUT2F6Z43LA5YMHROA73WH74GV7YVT4AC5ZZUXQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.297424198 (305h4m32.168507233s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.298110775 (305h4m32.16919382s) [noise=true , address=77.90.31.13:4627 , piece_id=QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.298944673 (305h4m32.170027718s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSB6JVIP6OQK2WFJPSHXVMIDS7MFMUAPXH6KL6L4CF3UKBYSYQZA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.299922345 (305h4m32.17100538s) [noise=true , address=77.90.31.14:4663 , piece_id=LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.300515594 (305h4m32.171598639s) [noise=true , address=145.79.227.29:4941 , piece_id=OQZLSACE5VUVAUPR5W2TTMHPTLLDYJV3LUQ2REZNY7YUMRG2PQRQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.301300189 (305h4m32.172383234s) [noise=true , address=145.79.227.20:4439 , piece_id=FVN6LPYNPLZ6WNNH3MYQK3LHKXERKUUYDPZYPAX5U3PWZSTF6LQA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.302242293 (305h4m32.173325338s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.302644288 (305h4m32.173727333s) [noise=true , address=145.79.227.14:4839 , piece_id=I2MDVDOJY7JAHN3YWEKIDETZAOPSGJK4CCQNULXC5R3VDXIFKT5Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.303913255 (305h4m32.1749963s) [noise=true , address=145.79.227.34:4905 , piece_id=SXWZTRLOU62WIGIDVMXEYVNM3LVXJNJC4Z5CEOVKXZFVLPI7KD2A, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.305187481 (305h4m32.176270526s) [noise=true , address=145.79.227.39:4015 , piece_id=5YJ6TIMERXKG5XCOZUNLNJLRFPIKARX3EEZ7NDL6YIBPSUZZEYHQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.306724738 (305h4m32.177807783s) [noise=true , address=145.79.227.31:4798 , piece_id=V4QMNVKADDFQPVT6FZBODVS5ZCYK524OWUVRDBU76GJNBDVA24CA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.309176456 (305h4m32.180259501s) [noise=true , address=145.79.227.36:4517 , piece_id=NKXOC3YTQA22MMKQSYVR7IJOOWPZSBJV3C3WW6ZUV3B7BFIWJJUA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:34.010761764 (305h4m33.88184481s) [upload=267496]: Next returned nil segment Dec 18 08:33:34.010780831 (305h4m33.881863896s) [upload=267496]: Waiting for error group managing segments... Dec 18 08:33:34.291265406 (305h4m34.162348471s) [noise=true , address=145.79.227.45:4231 , piece_id=TQHO636BRJK7JAFN2XIYPVTD47CQH4B5M342GLMCF7FBVAC2UH3Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.293371326 (305h4m34.164454381s) [noise=true , address=145.79.227.41:4085 , piece_id=X5OEVD562CONOGZJUA56KXI6UBN4GNS24IY4XUXSMLJENM5H3DQA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.295285591 (305h4m34.166368637s) [noise=true , address=145.79.227.14:4839 , piece_id=I2MDVDOJY7JAHN3YWEKIDETZAOPSGJK4CCQNULXC5R3VDXIFKT5Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.295870796 (305h4m34.166953841s) [noise=true , address=145.79.227.31:4798 , piece_id=V4QMNVKADDFQPVT6FZBODVS5ZCYK524OWUVRDBU76GJNBDVA24CA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.307409507 (305h4m34.178492582s) [noise=true , address=145.79.227.51:4411 , piece_id=EEHHLFQ6XNYKJQUT2F6Z43LA5YMHROA73WH74GV7YVT4AC5ZZUXQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.310409920 (305h4m34.181492965s) [noise=true , address=145.79.227.39:4015 , piece_id=5YJ6TIMERXKG5XCOZUNLNJLRFPIKARX3EEZ7NDL6YIBPSUZZEYHQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.315729651 (305h4m34.186812696s) [noise=true , address=145.79.227.34:4905 , piece_id=SXWZTRLOU62WIGIDVMXEYVNM3LVXJNJC4Z5CEOVKXZFVLPI7KD2A, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.317510652 (305h4m34.188593697s) [noise=true , address=145.79.227.36:4517 , piece_id=NKXOC3YTQA22MMKQSYVR7IJOOWPZSBJV3C3WW6ZUV3B7BFIWJJUA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.320827428 (305h4m34.191910473s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSB6JVIP6OQK2WFJPSHXVMIDS7MFMUAPXH6KL6L4CF3UKBYSYQZA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.374645999 (305h4m34.245729044s) [noise=true , address=145.79.227.20:4439 , piece_id=FVN6LPYNPLZ6WNNH3MYQK3LHKXERKUUYDPZYPAX5U3PWZSTF6LQA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.387615185 (305h4m34.25869823s) [noise=true , address=145.79.227.29:4941 , piece_id=OQZLSACE5VUVAUPR5W2TTMHPTLLDYJV3LUQ2REZNY7YUMRG2PQRQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.499570748 (305h4m34.370653793s) [noise=true , address=77.90.31.14:4663 , piece_id=LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.586131707 (305h4m34.457214752s) [noise=true , address=77.90.31.13:4627 , piece_id=QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.586162034 (305h4m34.45724508s) [seg_pos={0 0}, upload=267496]: Segment reached optimal threshold of 13 pieces. Dec 18 08:33:34.586441837 (305h4m34.457524892s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586451295 (305h4m34.45753434s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586829616 (305h4m34.457912671s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586850696 (305h4m34.457933741s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586885893 (305h4m34.457968938s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586896503 (305h4m34.457979538s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586946888 (305h4m34.458029924s) [seg_pos={0 0}, upload=267496]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:33:34.587023585 (305h4m34.45810663s) [upload=267496]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:33:34.587029046 (305h4m34.458112081s) [upload=267496]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:33:34.882748803 (305h4m34.753831858s) [upload=267496]: Done uploading segments... Dec 18 08:33:34.882773580 (305h4m34.753856625s) [upload=267496]: Upload finished. err: <nil> Duration: 15.815110386s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X...........X..X] duration: 30.003122127s trace: https://jaeger.dev.storj.io/trace/7e6640e8e766b46c FAIL (2025-12-18 08:34:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA) context deadline exceeded ======LOGS====== Dec 18 08:33:32.049014019 (305h4m31.920097054s) [upload=267496]: Starting upload Dec 18 08:33:32.049086377 (305h4m31.920169422s) [upload=267496]: Done starting upload Dec 18 08:33:32.049191306 (305h4m31.920274351s) [upload=267496]: Uploading segments... Dec 18 08:33:32.049419240 (305h4m31.920502285s) [upload=267496]: Got next segment. Inline: false Dec 18 08:33:32.049490996 (305h4m31.920574031s) [upload=267496]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:33:32.049497650 (305h4m31.920580685s) [upload=267496]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:33:32.295773565 (305h4m32.16685661s) [seg_pos={0 0}, upload=267496]: Begin upload segment... Dec 18 08:33:32.295846854 (305h4m32.166929889s) [seg_pos={0 0}, upload=267496]: Done begin upload segment. Dec 18 08:33:32.295885869 (305h4m32.166968904s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.295907280 (305h4m32.166990325s) [noise=true , address=145.79.227.41:4085 , piece_id=X5OEVD562CONOGZJUA56KXI6UBN4GNS24IY4XUXSMLJENM5H3DQA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.296542078 (305h4m32.167625113s) [noise=true , address=145.79.227.45:4231 , piece_id=TQHO636BRJK7JAFN2XIYPVTD47CQH4B5M342GLMCF7FBVAC2UH3Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.296984922 (305h4m32.168067967s) [noise=true , address=145.79.227.51:4411 , piece_id=EEHHLFQ6XNYKJQUT2F6Z43LA5YMHROA73WH74GV7YVT4AC5ZZUXQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.297424198 (305h4m32.168507233s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.298110775 (305h4m32.16919382s) [noise=true , address=77.90.31.13:4627 , piece_id=QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.298944673 (305h4m32.170027718s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSB6JVIP6OQK2WFJPSHXVMIDS7MFMUAPXH6KL6L4CF3UKBYSYQZA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.299922345 (305h4m32.17100538s) [noise=true , address=77.90.31.14:4663 , piece_id=LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.300515594 (305h4m32.171598639s) [noise=true , address=145.79.227.29:4941 , piece_id=OQZLSACE5VUVAUPR5W2TTMHPTLLDYJV3LUQ2REZNY7YUMRG2PQRQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.301300189 (305h4m32.172383234s) [noise=true , address=145.79.227.20:4439 , piece_id=FVN6LPYNPLZ6WNNH3MYQK3LHKXERKUUYDPZYPAX5U3PWZSTF6LQA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.302242293 (305h4m32.173325338s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.302644288 (305h4m32.173727333s) [noise=true , address=145.79.227.14:4839 , piece_id=I2MDVDOJY7JAHN3YWEKIDETZAOPSGJK4CCQNULXC5R3VDXIFKT5Q, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.303913255 (305h4m32.1749963s) [noise=true , address=145.79.227.34:4905 , piece_id=SXWZTRLOU62WIGIDVMXEYVNM3LVXJNJC4Z5CEOVKXZFVLPI7KD2A, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.305187481 (305h4m32.176270526s) [noise=true , address=145.79.227.39:4015 , piece_id=5YJ6TIMERXKG5XCOZUNLNJLRFPIKARX3EEZ7NDL6YIBPSUZZEYHQ, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.306724738 (305h4m32.177807783s) [noise=true , address=145.79.227.31:4798 , piece_id=V4QMNVKADDFQPVT6FZBODVS5ZCYK524OWUVRDBU76GJNBDVA24CA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:32.309176456 (305h4m32.180259501s) [noise=true , address=145.79.227.36:4517 , piece_id=NKXOC3YTQA22MMKQSYVR7IJOOWPZSBJV3C3WW6ZUV3B7BFIWJJUA, seg_pos={0 0}, upload=267496]: Uploading piece... Dec 18 08:33:34.010761764 (305h4m33.88184481s) [upload=267496]: Next returned nil segment Dec 18 08:33:34.010780831 (305h4m33.881863896s) [upload=267496]: Waiting for error group managing segments... Dec 18 08:33:34.291265406 (305h4m34.162348471s) [noise=true , address=145.79.227.45:4231 , piece_id=TQHO636BRJK7JAFN2XIYPVTD47CQH4B5M342GLMCF7FBVAC2UH3Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.293371326 (305h4m34.164454381s) [noise=true , address=145.79.227.41:4085 , piece_id=X5OEVD562CONOGZJUA56KXI6UBN4GNS24IY4XUXSMLJENM5H3DQA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.295285591 (305h4m34.166368637s) [noise=true , address=145.79.227.14:4839 , piece_id=I2MDVDOJY7JAHN3YWEKIDETZAOPSGJK4CCQNULXC5R3VDXIFKT5Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.295870796 (305h4m34.166953841s) [noise=true , address=145.79.227.31:4798 , piece_id=V4QMNVKADDFQPVT6FZBODVS5ZCYK524OWUVRDBU76GJNBDVA24CA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.307409507 (305h4m34.178492582s) [noise=true , address=145.79.227.51:4411 , piece_id=EEHHLFQ6XNYKJQUT2F6Z43LA5YMHROA73WH74GV7YVT4AC5ZZUXQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.310409920 (305h4m34.181492965s) [noise=true , address=145.79.227.39:4015 , piece_id=5YJ6TIMERXKG5XCOZUNLNJLRFPIKARX3EEZ7NDL6YIBPSUZZEYHQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.315729651 (305h4m34.186812696s) [noise=true , address=145.79.227.34:4905 , piece_id=SXWZTRLOU62WIGIDVMXEYVNM3LVXJNJC4Z5CEOVKXZFVLPI7KD2A, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.317510652 (305h4m34.188593697s) [noise=true , address=145.79.227.36:4517 , piece_id=NKXOC3YTQA22MMKQSYVR7IJOOWPZSBJV3C3WW6ZUV3B7BFIWJJUA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.320827428 (305h4m34.191910473s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSB6JVIP6OQK2WFJPSHXVMIDS7MFMUAPXH6KL6L4CF3UKBYSYQZA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.374645999 (305h4m34.245729044s) [noise=true , address=145.79.227.20:4439 , piece_id=FVN6LPYNPLZ6WNNH3MYQK3LHKXERKUUYDPZYPAX5U3PWZSTF6LQA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.387615185 (305h4m34.25869823s) [noise=true , address=145.79.227.29:4941 , piece_id=OQZLSACE5VUVAUPR5W2TTMHPTLLDYJV3LUQ2REZNY7YUMRG2PQRQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.499570748 (305h4m34.370653793s) [noise=true , address=77.90.31.14:4663 , piece_id=LQRBHKYZ3ROKG2CNR2GVEVAZVYPWDXGEI3H2HWHLL4EUAZTFUFAA, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.586131707 (305h4m34.457214752s) [noise=true , address=77.90.31.13:4627 , piece_id=QRH55OWN5ZIHZCQIPRGMLC3RCFOAUKXWFBM3VHGH2KZ4CXYBOOVQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: <nil> Dec 18 08:33:34.586162034 (305h4m34.45724508s) [seg_pos={0 0}, upload=267496]: Segment reached optimal threshold of 13 pieces. Dec 18 08:33:34.586441837 (305h4m34.457524892s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586451295 (305h4m34.45753434s) [noise=true , address=77.90.31.11:4555 , piece_id=T54CK6AK3OVSU67YDJP66KDP7RXUDSDEMFJ74GLEXB7N4SG2I6AQ, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586829616 (305h4m34.457912671s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586850696 (305h4m34.457933741s) [noise=true , address=45.91.116.13:4153 , piece_id=FWVWMMOGNUPY6ZDBZBLQZ3GXHPSKTYD62LTQVJ6ZJRMQ2YQIXKGQ, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586885893 (305h4m34.457968938s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 18 08:33:34.586896503 (305h4m34.457979538s) [noise=true , address=45.91.116.8:4045 , piece_id=DFL6FKEKBX2L74T2RQZLJRXM3EVQ4C7R266KF23DS4OYX5G74X7Q, seg_pos={0 0}, upload=267496]: Long tail cancellation due to optimal threshold Dec 18 08:33:34.586946888 (305h4m34.458029924s) [seg_pos={0 0}, upload=267496]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:33:34.587023585 (305h4m34.45810663s) [upload=267496]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:33:34.587029046 (305h4m34.458112081s) [upload=267496]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:33:34.882748803 (305h4m34.753831858s) [upload=267496]: Done uploading segments... Dec 18 08:33:34.882773580 (305h4m34.753856625s) [upload=267496]: Upload finished. err: <nil> Duration: 30.00314527s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X...........X..] duration: 28.731153123s trace: https://jaeger.dev.storj.io/trace/5192922ffff5f5e FAIL (2025-12-18 08:33:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 7SJFTDWV4UCOKEX56FXQHHZY7ZMR3BEOWNIT3ZQN66RIWKF6YHAQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UOWJQYXIE7ORERRM3SLAD2LLYKNVDFFL6RUMZ4WWXBYPFRKCOUAA) context deadline exceeded ======LOGS====== Dec 18 08:31:36.288485175 (305h2m36.15956822s) [upload=267468]: Starting upload Dec 18 08:31:36.288588091 (305h2m36.159671136s) [upload=267468]: Done starting upload Dec 18 08:31:36.288827607 (305h2m36.159910652s) [upload=267468]: Uploading segments... Dec 18 08:31:36.289100997 (305h2m36.160184043s) [upload=267468]: Got next segment. Inline: false Dec 18 08:31:36.289157685 (305h2m36.16024073s) [upload=267468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:31:36.289185718 (305h2m36.160268763s) [upload=267468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:31:36.543164632 (305h2m36.414247677s) [seg_pos={0 0}, upload=267468]: Begin upload segment... Dec 18 08:31:36.543257328 (305h2m36.414340373s) [seg_pos={0 0}, upload=267468]: Done begin upload segment. Dec 18 08:31:36.543285712 (305h2m36.414368757s) [noise=true , address=145.79.227.11:4731 , piece_id=UP3NOTXF2334CDKLYJVZPHAXJFLKCHGAJEHD7ELZASFJHP3WCPDQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.543357579 (305h2m36.414440624s) [noise=true , address=45.91.116.13:4155 , piece_id=S5U64CQRKN4JVAVDMK2V4MWW67C2VQRBZXF6KL3G2ZSSEZYTTKMQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.543907917 (305h2m36.414990962s) [noise=true , address=145.79.227.38:4977 , piece_id=2X4XAOJP3II3A6FXQRT6JWS7MGKG5E5HQVATKVIB4L3W6WKKFHLA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.544287740 (305h2m36.415370785s) [noise=true , address=145.79.227.16:4911 , piece_id=6OC5D4YUQWI5GOF6SJWFIPD3OABHLJGH46MAPMLK6TVXJEUDBPHQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.544302388 (305h2m36.415385433s) [noise=true , address=45.91.116.8:4048 , piece_id=7SJFTDWV4UCOKEX56FXQHHZY7ZMR3BEOWNIT3ZQN66RIWKF6YHAQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.544759098 (305h2m36.415842133s) [noise=true , address=145.79.227.48:4301 , piece_id=QV47G2EXH3RY6RPGIRVOD54F2JMKJ2NJSFQIFT7EY7R3IMSLQBHQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.544859630 (305h2m36.415942665s) [noise=true , address=145.79.227.46:4602 , piece_id=6OT7LGIVQYZQC3TVKKLB6CRXOFIF4IZXHA6WE27OXVWHW2V2ZOFA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.545138832 (305h2m36.416221877s) [noise=true , address=77.90.31.11:4553 , piece_id=4V24PZAHIJ6EAX4OWHOMM6AWWABPTWIN4YK3ASBJOHT42GHCIMJA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.545331668 (305h2m36.416414713s) [noise=true , address=45.91.116.4:4937 , piece_id=DKDXHIY3SNCFW5WL4SVJSDCBULCSOCEB7YJJZNV7KUDLGIWV5DAA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.545640276 (305h2m36.416723321s) [noise=true , address=45.91.116.12:4225 , piece_id=CLP6ZA2AHIRVAX7HMZ643YDQ55C7IFVJCH67WPF6CMKNO64KQ5IQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.545895311 (305h2m36.416978356s) [noise=true , address=77.90.31.17:4773 , piece_id=JCGQPD3FHIPUPEQD76X7O2LMOJZJIG7S7RSE2PN7AVAVNE3XXELQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.546288551 (305h2m36.417371596s) [noise=true , address=145.79.227.50:4374 , piece_id=VCL5BAVADVBJ6BSLBLD4AFQILICQP2AMBNKQSTJ277M6M2MNMWBA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.546736845 (305h2m36.41781988s) [noise=true , address=77.90.31.13:4627 , piece_id=DEFG4CG734UGY2JNQXXM7TQCRPXSQPQQ5NPL3YGWRZO45N6DTYSA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.547117489 (305h2m36.418200524s) [noise=true , address=77.90.31.10:4519 , piece_id=UOWJQYXIE7ORERRM3SLAD2LLYKNVDFFL6RUMZ4WWXBYPFRKCOUAA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.547125655 (305h2m36.41820869s) [noise=true , address=145.79.227.42:4122 , piece_id=ZYLSRZ6R53UCU6KTSTIPF2UIQ2KIK4NI2JGH3SDFLDYWUSZHHKYA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.547607593 (305h2m36.418690628s) [noise=true , address=145.79.227.47:4267 , piece_id=3BUEXFHKMHMZNF6HCP64EG2HGYSEPYSOJTT4QSROGHPJHAZWTL4A, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:37.979376809 (305h2m37.850459844s) [noise=true , address=77.90.31.11:4553 , piece_id=4V24PZAHIJ6EAX4OWHOMM6AWWABPTWIN4YK3ASBJOHT42GHCIMJA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 18 08:31:38.245491611 (305h2m38.116574646s) [upload=267468]: Next returned nil segment Dec 18 08:31:38.245508564 (305h2m38.116591629s) [upload=267468]: Waiting for error group managing segments... Dec 18 08:31:38.519572363 (305h2m38.390655398s) [noise=true , address=145.79.227.11:4731 , piece_id=UP3NOTXF2334CDKLYJVZPHAXJFLKCHGAJEHD7ELZASFJHP3WCPDQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.545643948 (305h2m38.416726993s) [noise=true , address=145.79.227.42:4122 , piece_id=ZYLSRZ6R53UCU6KTSTIPF2UIQ2KIK4NI2JGH3SDFLDYWUSZHHKYA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.549054352 (305h2m38.420137397s) [noise=true , address=145.79.227.47:4267 , piece_id=3BUEXFHKMHMZNF6HCP64EG2HGYSEPYSOJTT4QSROGHPJHAZWTL4A, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.555752336 (305h2m38.426835371s) [noise=true , address=145.79.227.38:4977 , piece_id=2X4XAOJP3II3A6FXQRT6JWS7MGKG5E5HQVATKVIB4L3W6WKKFHLA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.560225834 (305h2m38.431308869s) [noise=true , address=145.79.227.46:4602 , piece_id=6OT7LGIVQYZQC3TVKKLB6CRXOFIF4IZXHA6WE27OXVWHW2V2ZOFA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.562961965 (305h2m38.43404501s) [noise=true , address=145.79.227.50:4374 , piece_id=VCL5BAVADVBJ6BSLBLD4AFQILICQP2AMBNKQSTJ277M6M2MNMWBA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.600762083 (305h2m38.471845118s) [noise=true , address=145.79.227.48:4301 , piece_id=QV47G2EXH3RY6RPGIRVOD54F2JMKJ2NJSFQIFT7EY7R3IMSLQBHQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.637963541 (305h2m38.509046586s) [noise=true , address=145.79.227.16:4911 , piece_id=6OC5D4YUQWI5GOF6SJWFIPD3OABHLJGH46MAPMLK6TVXJEUDBPHQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.740880614 (305h2m38.611963659s) [noise=true , address=77.90.31.10:4519 , piece_id=UOWJQYXIE7ORERRM3SLAD2LLYKNVDFFL6RUMZ4WWXBYPFRKCOUAA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.782968949 (305h2m38.654051984s) [noise=true , address=77.90.31.13:4627 , piece_id=DEFG4CG734UGY2JNQXXM7TQCRPXSQPQQ5NPL3YGWRZO45N6DTYSA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.801373945 (305h2m38.67245699s) [noise=true , address=45.91.116.12:4225 , piece_id=CLP6ZA2AHIRVAX7HMZ643YDQ55C7IFVJCH67WPF6CMKNO64KQ5IQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.864803066 (305h2m38.735886111s) [noise=true , address=45.91.116.4:4937 , piece_id=DKDXHIY3SNCFW5WL4SVJSDCBULCSOCEB7YJJZNV7KUDLGIWV5DAA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.931943224 (305h2m38.803026259s) [noise=true , address=45.91.116.8:4048 , piece_id=7SJFTDWV4UCOKEX56FXQHHZY7ZMR3BEOWNIT3ZQN66RIWKF6YHAQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.931963944 (305h2m38.803046989s) [seg_pos={0 0}, upload=267468]: Segment reached optimal threshold of 13 pieces. Dec 18 08:31:38.932166118 (305h2m38.803249163s) [noise=true , address=77.90.31.17:4773 , piece_id=JCGQPD3FHIPUPEQD76X7O2LMOJZJIG7S7RSE2PN7AVAVNE3XXELQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 18 08:31:38.932175236 (305h2m38.803258281s) [noise=true , address=77.90.31.17:4773 , piece_id=JCGQPD3FHIPUPEQD76X7O2LMOJZJIG7S7RSE2PN7AVAVNE3XXELQ, seg_pos={0 0}, upload=267468]: Long tail cancellation due to optimal threshold Dec 18 08:31:38.932201336 (305h2m38.803284381s) [noise=true , address=45.91.116.13:4155 , piece_id=S5U64CQRKN4JVAVDMK2V4MWW67C2VQRBZXF6KL3G2ZSSEZYTTKMQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 18 08:31:38.932210523 (305h2m38.803293568s) [noise=true , address=45.91.116.13:4155 , piece_id=S5U64CQRKN4JVAVDMK2V4MWW67C2VQRBZXF6KL3G2ZSSEZYTTKMQ, seg_pos={0 0}, upload=267468]: Long tail cancellation due to optimal threshold Dec 18 08:31:38.932240961 (305h2m38.803324006s) [seg_pos={0 0}, upload=267468]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:31:38.932325502 (305h2m38.803408537s) [upload=267468]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:31:38.932330652 (305h2m38.803413697s) [upload=267468]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:31:39.235620602 (305h2m39.106703658s) [upload=267468]: Done uploading segments... Dec 18 08:31:39.235644508 (305h2m39.106727563s) [upload=267468]: Upload finished. err: <nil> Duration: 28.731186025s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X...........X.] duration: 6.093924327s trace: https://jaeger.dev.storj.io/trace/5192922ffff5f5e FAIL (2025-12-18 08:32:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 7SJFTDWV4UCOKEX56FXQHHZY7ZMR3BEOWNIT3ZQN66RIWKF6YHAQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UOWJQYXIE7ORERRM3SLAD2LLYKNVDFFL6RUMZ4WWXBYPFRKCOUAA) context deadline exceeded ======LOGS====== Dec 18 08:31:36.288485175 (305h2m36.15956822s) [upload=267468]: Starting upload Dec 18 08:31:36.288588091 (305h2m36.159671136s) [upload=267468]: Done starting upload Dec 18 08:31:36.288827607 (305h2m36.159910652s) [upload=267468]: Uploading segments... Dec 18 08:31:36.289100997 (305h2m36.160184043s) [upload=267468]: Got next segment. Inline: false Dec 18 08:31:36.289157685 (305h2m36.16024073s) [upload=267468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:31:36.289185718 (305h2m36.160268763s) [upload=267468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:31:36.543164632 (305h2m36.414247677s) [seg_pos={0 0}, upload=267468]: Begin upload segment... Dec 18 08:31:36.543257328 (305h2m36.414340373s) [seg_pos={0 0}, upload=267468]: Done begin upload segment. Dec 18 08:31:36.543285712 (305h2m36.414368757s) [noise=true , address=145.79.227.11:4731 , piece_id=UP3NOTXF2334CDKLYJVZPHAXJFLKCHGAJEHD7ELZASFJHP3WCPDQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.543357579 (305h2m36.414440624s) [noise=true , address=45.91.116.13:4155 , piece_id=S5U64CQRKN4JVAVDMK2V4MWW67C2VQRBZXF6KL3G2ZSSEZYTTKMQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.543907917 (305h2m36.414990962s) [noise=true , address=145.79.227.38:4977 , piece_id=2X4XAOJP3II3A6FXQRT6JWS7MGKG5E5HQVATKVIB4L3W6WKKFHLA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.544287740 (305h2m36.415370785s) [noise=true , address=145.79.227.16:4911 , piece_id=6OC5D4YUQWI5GOF6SJWFIPD3OABHLJGH46MAPMLK6TVXJEUDBPHQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.544302388 (305h2m36.415385433s) [noise=true , address=45.91.116.8:4048 , piece_id=7SJFTDWV4UCOKEX56FXQHHZY7ZMR3BEOWNIT3ZQN66RIWKF6YHAQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.544759098 (305h2m36.415842133s) [noise=true , address=145.79.227.48:4301 , piece_id=QV47G2EXH3RY6RPGIRVOD54F2JMKJ2NJSFQIFT7EY7R3IMSLQBHQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.544859630 (305h2m36.415942665s) [noise=true , address=145.79.227.46:4602 , piece_id=6OT7LGIVQYZQC3TVKKLB6CRXOFIF4IZXHA6WE27OXVWHW2V2ZOFA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.545138832 (305h2m36.416221877s) [noise=true , address=77.90.31.11:4553 , piece_id=4V24PZAHIJ6EAX4OWHOMM6AWWABPTWIN4YK3ASBJOHT42GHCIMJA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.545331668 (305h2m36.416414713s) [noise=true , address=45.91.116.4:4937 , piece_id=DKDXHIY3SNCFW5WL4SVJSDCBULCSOCEB7YJJZNV7KUDLGIWV5DAA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.545640276 (305h2m36.416723321s) [noise=true , address=45.91.116.12:4225 , piece_id=CLP6ZA2AHIRVAX7HMZ643YDQ55C7IFVJCH67WPF6CMKNO64KQ5IQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.545895311 (305h2m36.416978356s) [noise=true , address=77.90.31.17:4773 , piece_id=JCGQPD3FHIPUPEQD76X7O2LMOJZJIG7S7RSE2PN7AVAVNE3XXELQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.546288551 (305h2m36.417371596s) [noise=true , address=145.79.227.50:4374 , piece_id=VCL5BAVADVBJ6BSLBLD4AFQILICQP2AMBNKQSTJ277M6M2MNMWBA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.546736845 (305h2m36.41781988s) [noise=true , address=77.90.31.13:4627 , piece_id=DEFG4CG734UGY2JNQXXM7TQCRPXSQPQQ5NPL3YGWRZO45N6DTYSA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.547117489 (305h2m36.418200524s) [noise=true , address=77.90.31.10:4519 , piece_id=UOWJQYXIE7ORERRM3SLAD2LLYKNVDFFL6RUMZ4WWXBYPFRKCOUAA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.547125655 (305h2m36.41820869s) [noise=true , address=145.79.227.42:4122 , piece_id=ZYLSRZ6R53UCU6KTSTIPF2UIQ2KIK4NI2JGH3SDFLDYWUSZHHKYA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.547607593 (305h2m36.418690628s) [noise=true , address=145.79.227.47:4267 , piece_id=3BUEXFHKMHMZNF6HCP64EG2HGYSEPYSOJTT4QSROGHPJHAZWTL4A, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:37.979376809 (305h2m37.850459844s) [noise=true , address=77.90.31.11:4553 , piece_id=4V24PZAHIJ6EAX4OWHOMM6AWWABPTWIN4YK3ASBJOHT42GHCIMJA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 18 08:31:38.245491611 (305h2m38.116574646s) [upload=267468]: Next returned nil segment Dec 18 08:31:38.245508564 (305h2m38.116591629s) [upload=267468]: Waiting for error group managing segments... Dec 18 08:31:38.519572363 (305h2m38.390655398s) [noise=true , address=145.79.227.11:4731 , piece_id=UP3NOTXF2334CDKLYJVZPHAXJFLKCHGAJEHD7ELZASFJHP3WCPDQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.545643948 (305h2m38.416726993s) [noise=true , address=145.79.227.42:4122 , piece_id=ZYLSRZ6R53UCU6KTSTIPF2UIQ2KIK4NI2JGH3SDFLDYWUSZHHKYA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.549054352 (305h2m38.420137397s) [noise=true , address=145.79.227.47:4267 , piece_id=3BUEXFHKMHMZNF6HCP64EG2HGYSEPYSOJTT4QSROGHPJHAZWTL4A, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.555752336 (305h2m38.426835371s) [noise=true , address=145.79.227.38:4977 , piece_id=2X4XAOJP3II3A6FXQRT6JWS7MGKG5E5HQVATKVIB4L3W6WKKFHLA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.560225834 (305h2m38.431308869s) [noise=true , address=145.79.227.46:4602 , piece_id=6OT7LGIVQYZQC3TVKKLB6CRXOFIF4IZXHA6WE27OXVWHW2V2ZOFA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.562961965 (305h2m38.43404501s) [noise=true , address=145.79.227.50:4374 , piece_id=VCL5BAVADVBJ6BSLBLD4AFQILICQP2AMBNKQSTJ277M6M2MNMWBA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.600762083 (305h2m38.471845118s) [noise=true , address=145.79.227.48:4301 , piece_id=QV47G2EXH3RY6RPGIRVOD54F2JMKJ2NJSFQIFT7EY7R3IMSLQBHQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.637963541 (305h2m38.509046586s) [noise=true , address=145.79.227.16:4911 , piece_id=6OC5D4YUQWI5GOF6SJWFIPD3OABHLJGH46MAPMLK6TVXJEUDBPHQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.740880614 (305h2m38.611963659s) [noise=true , address=77.90.31.10:4519 , piece_id=UOWJQYXIE7ORERRM3SLAD2LLYKNVDFFL6RUMZ4WWXBYPFRKCOUAA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.782968949 (305h2m38.654051984s) [noise=true , address=77.90.31.13:4627 , piece_id=DEFG4CG734UGY2JNQXXM7TQCRPXSQPQQ5NPL3YGWRZO45N6DTYSA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.801373945 (305h2m38.67245699s) [noise=true , address=45.91.116.12:4225 , piece_id=CLP6ZA2AHIRVAX7HMZ643YDQ55C7IFVJCH67WPF6CMKNO64KQ5IQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.864803066 (305h2m38.735886111s) [noise=true , address=45.91.116.4:4937 , piece_id=DKDXHIY3SNCFW5WL4SVJSDCBULCSOCEB7YJJZNV7KUDLGIWV5DAA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.931943224 (305h2m38.803026259s) [noise=true , address=45.91.116.8:4048 , piece_id=7SJFTDWV4UCOKEX56FXQHHZY7ZMR3BEOWNIT3ZQN66RIWKF6YHAQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.931963944 (305h2m38.803046989s) [seg_pos={0 0}, upload=267468]: Segment reached optimal threshold of 13 pieces. Dec 18 08:31:38.932166118 (305h2m38.803249163s) [noise=true , address=77.90.31.17:4773 , piece_id=JCGQPD3FHIPUPEQD76X7O2LMOJZJIG7S7RSE2PN7AVAVNE3XXELQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 18 08:31:38.932175236 (305h2m38.803258281s) [noise=true , address=77.90.31.17:4773 , piece_id=JCGQPD3FHIPUPEQD76X7O2LMOJZJIG7S7RSE2PN7AVAVNE3XXELQ, seg_pos={0 0}, upload=267468]: Long tail cancellation due to optimal threshold Dec 18 08:31:38.932201336 (305h2m38.803284381s) [noise=true , address=45.91.116.13:4155 , piece_id=S5U64CQRKN4JVAVDMK2V4MWW67C2VQRBZXF6KL3G2ZSSEZYTTKMQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 18 08:31:38.932210523 (305h2m38.803293568s) [noise=true , address=45.91.116.13:4155 , piece_id=S5U64CQRKN4JVAVDMK2V4MWW67C2VQRBZXF6KL3G2ZSSEZYTTKMQ, seg_pos={0 0}, upload=267468]: Long tail cancellation due to optimal threshold Dec 18 08:31:38.932240961 (305h2m38.803324006s) [seg_pos={0 0}, upload=267468]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:31:38.932325502 (305h2m38.803408537s) [upload=267468]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:31:38.932330652 (305h2m38.803413697s) [upload=267468]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:31:39.235620602 (305h2m39.106703658s) [upload=267468]: Done uploading segments... Dec 18 08:31:39.235644508 (305h2m39.106727563s) [upload=267468]: Upload finished. err: <nil> Duration: 6.093946409s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...........X] duration: 30.0025208s trace: https://jaeger.dev.storj.io/trace/5192922ffff5f5e FAIL (2025-12-18 08:32:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 7SJFTDWV4UCOKEX56FXQHHZY7ZMR3BEOWNIT3ZQN66RIWKF6YHAQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UOWJQYXIE7ORERRM3SLAD2LLYKNVDFFL6RUMZ4WWXBYPFRKCOUAA) context deadline exceeded ======LOGS====== Dec 18 08:31:36.288485175 (305h2m36.15956822s) [upload=267468]: Starting upload Dec 18 08:31:36.288588091 (305h2m36.159671136s) [upload=267468]: Done starting upload Dec 18 08:31:36.288827607 (305h2m36.159910652s) [upload=267468]: Uploading segments... Dec 18 08:31:36.289100997 (305h2m36.160184043s) [upload=267468]: Got next segment. Inline: false Dec 18 08:31:36.289157685 (305h2m36.16024073s) [upload=267468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:31:36.289185718 (305h2m36.160268763s) [upload=267468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:31:36.543164632 (305h2m36.414247677s) [seg_pos={0 0}, upload=267468]: Begin upload segment... Dec 18 08:31:36.543257328 (305h2m36.414340373s) [seg_pos={0 0}, upload=267468]: Done begin upload segment. Dec 18 08:31:36.543285712 (305h2m36.414368757s) [noise=true , address=145.79.227.11:4731 , piece_id=UP3NOTXF2334CDKLYJVZPHAXJFLKCHGAJEHD7ELZASFJHP3WCPDQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.543357579 (305h2m36.414440624s) [noise=true , address=45.91.116.13:4155 , piece_id=S5U64CQRKN4JVAVDMK2V4MWW67C2VQRBZXF6KL3G2ZSSEZYTTKMQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.543907917 (305h2m36.414990962s) [noise=true , address=145.79.227.38:4977 , piece_id=2X4XAOJP3II3A6FXQRT6JWS7MGKG5E5HQVATKVIB4L3W6WKKFHLA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.544287740 (305h2m36.415370785s) [noise=true , address=145.79.227.16:4911 , piece_id=6OC5D4YUQWI5GOF6SJWFIPD3OABHLJGH46MAPMLK6TVXJEUDBPHQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.544302388 (305h2m36.415385433s) [noise=true , address=45.91.116.8:4048 , piece_id=7SJFTDWV4UCOKEX56FXQHHZY7ZMR3BEOWNIT3ZQN66RIWKF6YHAQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.544759098 (305h2m36.415842133s) [noise=true , address=145.79.227.48:4301 , piece_id=QV47G2EXH3RY6RPGIRVOD54F2JMKJ2NJSFQIFT7EY7R3IMSLQBHQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.544859630 (305h2m36.415942665s) [noise=true , address=145.79.227.46:4602 , piece_id=6OT7LGIVQYZQC3TVKKLB6CRXOFIF4IZXHA6WE27OXVWHW2V2ZOFA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.545138832 (305h2m36.416221877s) [noise=true , address=77.90.31.11:4553 , piece_id=4V24PZAHIJ6EAX4OWHOMM6AWWABPTWIN4YK3ASBJOHT42GHCIMJA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.545331668 (305h2m36.416414713s) [noise=true , address=45.91.116.4:4937 , piece_id=DKDXHIY3SNCFW5WL4SVJSDCBULCSOCEB7YJJZNV7KUDLGIWV5DAA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.545640276 (305h2m36.416723321s) [noise=true , address=45.91.116.12:4225 , piece_id=CLP6ZA2AHIRVAX7HMZ643YDQ55C7IFVJCH67WPF6CMKNO64KQ5IQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.545895311 (305h2m36.416978356s) [noise=true , address=77.90.31.17:4773 , piece_id=JCGQPD3FHIPUPEQD76X7O2LMOJZJIG7S7RSE2PN7AVAVNE3XXELQ, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.546288551 (305h2m36.417371596s) [noise=true , address=145.79.227.50:4374 , piece_id=VCL5BAVADVBJ6BSLBLD4AFQILICQP2AMBNKQSTJ277M6M2MNMWBA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.546736845 (305h2m36.41781988s) [noise=true , address=77.90.31.13:4627 , piece_id=DEFG4CG734UGY2JNQXXM7TQCRPXSQPQQ5NPL3YGWRZO45N6DTYSA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.547117489 (305h2m36.418200524s) [noise=true , address=77.90.31.10:4519 , piece_id=UOWJQYXIE7ORERRM3SLAD2LLYKNVDFFL6RUMZ4WWXBYPFRKCOUAA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.547125655 (305h2m36.41820869s) [noise=true , address=145.79.227.42:4122 , piece_id=ZYLSRZ6R53UCU6KTSTIPF2UIQ2KIK4NI2JGH3SDFLDYWUSZHHKYA, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:36.547607593 (305h2m36.418690628s) [noise=true , address=145.79.227.47:4267 , piece_id=3BUEXFHKMHMZNF6HCP64EG2HGYSEPYSOJTT4QSROGHPJHAZWTL4A, seg_pos={0 0}, upload=267468]: Uploading piece... Dec 18 08:31:37.979376809 (305h2m37.850459844s) [noise=true , address=77.90.31.11:4553 , piece_id=4V24PZAHIJ6EAX4OWHOMM6AWWABPTWIN4YK3ASBJOHT42GHCIMJA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 18 08:31:38.245491611 (305h2m38.116574646s) [upload=267468]: Next returned nil segment Dec 18 08:31:38.245508564 (305h2m38.116591629s) [upload=267468]: Waiting for error group managing segments... Dec 18 08:31:38.519572363 (305h2m38.390655398s) [noise=true , address=145.79.227.11:4731 , piece_id=UP3NOTXF2334CDKLYJVZPHAXJFLKCHGAJEHD7ELZASFJHP3WCPDQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.545643948 (305h2m38.416726993s) [noise=true , address=145.79.227.42:4122 , piece_id=ZYLSRZ6R53UCU6KTSTIPF2UIQ2KIK4NI2JGH3SDFLDYWUSZHHKYA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.549054352 (305h2m38.420137397s) [noise=true , address=145.79.227.47:4267 , piece_id=3BUEXFHKMHMZNF6HCP64EG2HGYSEPYSOJTT4QSROGHPJHAZWTL4A, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.555752336 (305h2m38.426835371s) [noise=true , address=145.79.227.38:4977 , piece_id=2X4XAOJP3II3A6FXQRT6JWS7MGKG5E5HQVATKVIB4L3W6WKKFHLA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.560225834 (305h2m38.431308869s) [noise=true , address=145.79.227.46:4602 , piece_id=6OT7LGIVQYZQC3TVKKLB6CRXOFIF4IZXHA6WE27OXVWHW2V2ZOFA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.562961965 (305h2m38.43404501s) [noise=true , address=145.79.227.50:4374 , piece_id=VCL5BAVADVBJ6BSLBLD4AFQILICQP2AMBNKQSTJ277M6M2MNMWBA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.600762083 (305h2m38.471845118s) [noise=true , address=145.79.227.48:4301 , piece_id=QV47G2EXH3RY6RPGIRVOD54F2JMKJ2NJSFQIFT7EY7R3IMSLQBHQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.637963541 (305h2m38.509046586s) [noise=true , address=145.79.227.16:4911 , piece_id=6OC5D4YUQWI5GOF6SJWFIPD3OABHLJGH46MAPMLK6TVXJEUDBPHQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.740880614 (305h2m38.611963659s) [noise=true , address=77.90.31.10:4519 , piece_id=UOWJQYXIE7ORERRM3SLAD2LLYKNVDFFL6RUMZ4WWXBYPFRKCOUAA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.782968949 (305h2m38.654051984s) [noise=true , address=77.90.31.13:4627 , piece_id=DEFG4CG734UGY2JNQXXM7TQCRPXSQPQQ5NPL3YGWRZO45N6DTYSA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.801373945 (305h2m38.67245699s) [noise=true , address=45.91.116.12:4225 , piece_id=CLP6ZA2AHIRVAX7HMZ643YDQ55C7IFVJCH67WPF6CMKNO64KQ5IQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.864803066 (305h2m38.735886111s) [noise=true , address=45.91.116.4:4937 , piece_id=DKDXHIY3SNCFW5WL4SVJSDCBULCSOCEB7YJJZNV7KUDLGIWV5DAA, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.931943224 (305h2m38.803026259s) [noise=true , address=45.91.116.8:4048 , piece_id=7SJFTDWV4UCOKEX56FXQHHZY7ZMR3BEOWNIT3ZQN66RIWKF6YHAQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: <nil> Dec 18 08:31:38.931963944 (305h2m38.803046989s) [seg_pos={0 0}, upload=267468]: Segment reached optimal threshold of 13 pieces. Dec 18 08:31:38.932166118 (305h2m38.803249163s) [noise=true , address=77.90.31.17:4773 , piece_id=JCGQPD3FHIPUPEQD76X7O2LMOJZJIG7S7RSE2PN7AVAVNE3XXELQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 18 08:31:38.932175236 (305h2m38.803258281s) [noise=true , address=77.90.31.17:4773 , piece_id=JCGQPD3FHIPUPEQD76X7O2LMOJZJIG7S7RSE2PN7AVAVNE3XXELQ, seg_pos={0 0}, upload=267468]: Long tail cancellation due to optimal threshold Dec 18 08:31:38.932201336 (305h2m38.803284381s) [noise=true , address=45.91.116.13:4155 , piece_id=S5U64CQRKN4JVAVDMK2V4MWW67C2VQRBZXF6KL3G2ZSSEZYTTKMQ, seg_pos={0 0}, upload=267468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 18 08:31:38.932210523 (305h2m38.803293568s) [noise=true , address=45.91.116.13:4155 , piece_id=S5U64CQRKN4JVAVDMK2V4MWW67C2VQRBZXF6KL3G2ZSSEZYTTKMQ, seg_pos={0 0}, upload=267468]: Long tail cancellation due to optimal threshold Dec 18 08:31:38.932240961 (305h2m38.803324006s) [seg_pos={0 0}, upload=267468]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:31:38.932325502 (305h2m38.803408537s) [upload=267468]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:31:38.932330652 (305h2m38.803413697s) [upload=267468]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:31:39.235620602 (305h2m39.106703658s) [upload=267468]: Done uploading segments... Dec 18 08:31:39.235644508 (305h2m39.106727563s) [upload=267468]: Upload finished. err: <nil> Duration: 30.002543063s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.764091139s trace: https://jaeger.dev.storj.io/trace/162d3bebbccb0cd4 FAIL (2025-12-18 08:30:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 08:21:52.979634680 (304h52m52.850717715s) [upload=267329]: Starting upload Dec 18 08:21:52.979689916 (304h52m52.850772961s) [upload=267329]: Done starting upload Dec 18 08:21:52.979868376 (304h52m52.850951421s) [upload=267329]: Uploading segments... Dec 18 08:21:52.979915966 (304h52m52.850999001s) [upload=267329]: Got next segment. Inline: false Dec 18 08:21:52.979939922 (304h52m52.851022957s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:21:52.979943899 (304h52m52.851026944s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:21:53.227436064 (304h52m53.09851911s) [seg_pos={0 0}, upload=267329]: Begin upload segment... Dec 18 08:21:53.227511688 (304h52m53.098594723s) [seg_pos={0 0}, upload=267329]: Done begin upload segment. Dec 18 08:21:53.227584286 (304h52m53.098667321s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.229933821 (304h52m53.101016866s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.230418473 (304h52m53.101501508s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231688963 (304h52m53.102771998s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231930132 (304h52m53.103013167s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.232747668 (304h52m53.103830703s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.233027391 (304h52m53.104110426s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234000514 (304h52m53.105083549s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234145360 (304h52m53.105228395s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235434395 (304h52m53.10651744s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235584220 (304h52m53.106667255s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.236732076 (304h52m53.107815111s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.237519464 (304h52m53.108602499s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.239600678 (304h52m53.110683723s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.240042359 (304h52m53.111125404s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.242933985 (304h52m53.11401702s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:54.851541589 (304h52m54.722624634s) [upload=267329]: Next returned nil segment Dec 18 08:21:54.851554333 (304h52m54.722637378s) [upload=267329]: Waiting for error group managing segments... Dec 18 08:21:55.235427483 (304h52m55.106510538s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236089233 (304h52m55.107172278s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236667895 (304h52m55.10775094s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.252488160 (304h52m55.123571195s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255770450 (304h52m55.126853495s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255998223 (304h52m55.127081258s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.256221949 (304h52m55.127304985s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.258454531 (304h52m55.129537566s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.262083320 (304h52m55.133166355s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.302758765 (304h52m55.17384181s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.303381982 (304h52m55.174465027s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.437044621 (304h52m55.308127666s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.476998842 (304h52m55.348081877s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.477023177 (304h52m55.348106222s) [seg_pos={0 0}, upload=267329]: Segment reached optimal threshold of 13 pieces. Dec 18 08:21:55.477226615 (304h52m55.34830965s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477234159 (304h52m55.348317204s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477517129 (304h52m55.348600174s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477544811 (304h52m55.348627856s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477714985 (304h52m55.34879803s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477722289 (304h52m55.348805334s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477740153 (304h52m55.348823198s) [seg_pos={0 0}, upload=267329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:21:55.477862617 (304h52m55.348945662s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:21:55.477867526 (304h52m55.348950561s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:21:55.773001729 (304h52m55.644084784s) [upload=267329]: Done uploading segments... Dec 18 08:21:55.773021757 (304h52m55.644104802s) [upload=267329]: Upload finished. err: <nil> Duration: 5.764124883s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.172082432s trace: https://jaeger.dev.storj.io/trace/162d3bebbccb0cd4 FAIL (2025-12-18 08:29:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 08:21:52.979634680 (304h52m52.850717715s) [upload=267329]: Starting upload Dec 18 08:21:52.979689916 (304h52m52.850772961s) [upload=267329]: Done starting upload Dec 18 08:21:52.979868376 (304h52m52.850951421s) [upload=267329]: Uploading segments... Dec 18 08:21:52.979915966 (304h52m52.850999001s) [upload=267329]: Got next segment. Inline: false Dec 18 08:21:52.979939922 (304h52m52.851022957s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:21:52.979943899 (304h52m52.851026944s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:21:53.227436064 (304h52m53.09851911s) [seg_pos={0 0}, upload=267329]: Begin upload segment... Dec 18 08:21:53.227511688 (304h52m53.098594723s) [seg_pos={0 0}, upload=267329]: Done begin upload segment. Dec 18 08:21:53.227584286 (304h52m53.098667321s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.229933821 (304h52m53.101016866s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.230418473 (304h52m53.101501508s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231688963 (304h52m53.102771998s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231930132 (304h52m53.103013167s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.232747668 (304h52m53.103830703s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.233027391 (304h52m53.104110426s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234000514 (304h52m53.105083549s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234145360 (304h52m53.105228395s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235434395 (304h52m53.10651744s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235584220 (304h52m53.106667255s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.236732076 (304h52m53.107815111s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.237519464 (304h52m53.108602499s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.239600678 (304h52m53.110683723s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.240042359 (304h52m53.111125404s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.242933985 (304h52m53.11401702s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:54.851541589 (304h52m54.722624634s) [upload=267329]: Next returned nil segment Dec 18 08:21:54.851554333 (304h52m54.722637378s) [upload=267329]: Waiting for error group managing segments... Dec 18 08:21:55.235427483 (304h52m55.106510538s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236089233 (304h52m55.107172278s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236667895 (304h52m55.10775094s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.252488160 (304h52m55.123571195s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255770450 (304h52m55.126853495s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255998223 (304h52m55.127081258s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.256221949 (304h52m55.127304985s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.258454531 (304h52m55.129537566s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.262083320 (304h52m55.133166355s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.302758765 (304h52m55.17384181s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.303381982 (304h52m55.174465027s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.437044621 (304h52m55.308127666s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.476998842 (304h52m55.348081877s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.477023177 (304h52m55.348106222s) [seg_pos={0 0}, upload=267329]: Segment reached optimal threshold of 13 pieces. Dec 18 08:21:55.477226615 (304h52m55.34830965s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477234159 (304h52m55.348317204s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477517129 (304h52m55.348600174s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477544811 (304h52m55.348627856s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477714985 (304h52m55.34879803s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477722289 (304h52m55.348805334s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477740153 (304h52m55.348823198s) [seg_pos={0 0}, upload=267329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:21:55.477862617 (304h52m55.348945662s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:21:55.477867526 (304h52m55.348950561s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:21:55.773001729 (304h52m55.644084784s) [upload=267329]: Done uploading segments... Dec 18 08:21:55.773021757 (304h52m55.644104802s) [upload=267329]: Upload finished. err: <nil> Duration: 6.172098043s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.802854367s trace: https://jaeger.dev.storj.io/trace/162d3bebbccb0cd4 FAIL (2025-12-18 08:28:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 08:21:52.979634680 (304h52m52.850717715s) [upload=267329]: Starting upload Dec 18 08:21:52.979689916 (304h52m52.850772961s) [upload=267329]: Done starting upload Dec 18 08:21:52.979868376 (304h52m52.850951421s) [upload=267329]: Uploading segments... Dec 18 08:21:52.979915966 (304h52m52.850999001s) [upload=267329]: Got next segment. Inline: false Dec 18 08:21:52.979939922 (304h52m52.851022957s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:21:52.979943899 (304h52m52.851026944s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:21:53.227436064 (304h52m53.09851911s) [seg_pos={0 0}, upload=267329]: Begin upload segment... Dec 18 08:21:53.227511688 (304h52m53.098594723s) [seg_pos={0 0}, upload=267329]: Done begin upload segment. Dec 18 08:21:53.227584286 (304h52m53.098667321s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.229933821 (304h52m53.101016866s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.230418473 (304h52m53.101501508s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231688963 (304h52m53.102771998s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231930132 (304h52m53.103013167s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.232747668 (304h52m53.103830703s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.233027391 (304h52m53.104110426s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234000514 (304h52m53.105083549s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234145360 (304h52m53.105228395s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235434395 (304h52m53.10651744s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235584220 (304h52m53.106667255s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.236732076 (304h52m53.107815111s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.237519464 (304h52m53.108602499s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.239600678 (304h52m53.110683723s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.240042359 (304h52m53.111125404s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.242933985 (304h52m53.11401702s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:54.851541589 (304h52m54.722624634s) [upload=267329]: Next returned nil segment Dec 18 08:21:54.851554333 (304h52m54.722637378s) [upload=267329]: Waiting for error group managing segments... Dec 18 08:21:55.235427483 (304h52m55.106510538s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236089233 (304h52m55.107172278s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236667895 (304h52m55.10775094s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.252488160 (304h52m55.123571195s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255770450 (304h52m55.126853495s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255998223 (304h52m55.127081258s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.256221949 (304h52m55.127304985s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.258454531 (304h52m55.129537566s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.262083320 (304h52m55.133166355s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.302758765 (304h52m55.17384181s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.303381982 (304h52m55.174465027s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.437044621 (304h52m55.308127666s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.476998842 (304h52m55.348081877s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.477023177 (304h52m55.348106222s) [seg_pos={0 0}, upload=267329]: Segment reached optimal threshold of 13 pieces. Dec 18 08:21:55.477226615 (304h52m55.34830965s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477234159 (304h52m55.348317204s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477517129 (304h52m55.348600174s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477544811 (304h52m55.348627856s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477714985 (304h52m55.34879803s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477722289 (304h52m55.348805334s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477740153 (304h52m55.348823198s) [seg_pos={0 0}, upload=267329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:21:55.477862617 (304h52m55.348945662s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:21:55.477867526 (304h52m55.348950561s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:21:55.773001729 (304h52m55.644084784s) [upload=267329]: Done uploading segments... Dec 18 08:21:55.773021757 (304h52m55.644104802s) [upload=267329]: Upload finished. err: <nil> Duration: 5.802886008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.181810695s trace: https://jaeger.dev.storj.io/trace/162d3bebbccb0cd4 FAIL (2025-12-18 08:27:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 08:21:52.979634680 (304h52m52.850717715s) [upload=267329]: Starting upload Dec 18 08:21:52.979689916 (304h52m52.850772961s) [upload=267329]: Done starting upload Dec 18 08:21:52.979868376 (304h52m52.850951421s) [upload=267329]: Uploading segments... Dec 18 08:21:52.979915966 (304h52m52.850999001s) [upload=267329]: Got next segment. Inline: false Dec 18 08:21:52.979939922 (304h52m52.851022957s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:21:52.979943899 (304h52m52.851026944s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:21:53.227436064 (304h52m53.09851911s) [seg_pos={0 0}, upload=267329]: Begin upload segment... Dec 18 08:21:53.227511688 (304h52m53.098594723s) [seg_pos={0 0}, upload=267329]: Done begin upload segment. Dec 18 08:21:53.227584286 (304h52m53.098667321s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.229933821 (304h52m53.101016866s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.230418473 (304h52m53.101501508s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231688963 (304h52m53.102771998s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231930132 (304h52m53.103013167s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.232747668 (304h52m53.103830703s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.233027391 (304h52m53.104110426s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234000514 (304h52m53.105083549s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234145360 (304h52m53.105228395s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235434395 (304h52m53.10651744s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235584220 (304h52m53.106667255s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.236732076 (304h52m53.107815111s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.237519464 (304h52m53.108602499s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.239600678 (304h52m53.110683723s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.240042359 (304h52m53.111125404s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.242933985 (304h52m53.11401702s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:54.851541589 (304h52m54.722624634s) [upload=267329]: Next returned nil segment Dec 18 08:21:54.851554333 (304h52m54.722637378s) [upload=267329]: Waiting for error group managing segments... Dec 18 08:21:55.235427483 (304h52m55.106510538s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236089233 (304h52m55.107172278s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236667895 (304h52m55.10775094s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.252488160 (304h52m55.123571195s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255770450 (304h52m55.126853495s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255998223 (304h52m55.127081258s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.256221949 (304h52m55.127304985s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.258454531 (304h52m55.129537566s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.262083320 (304h52m55.133166355s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.302758765 (304h52m55.17384181s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.303381982 (304h52m55.174465027s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.437044621 (304h52m55.308127666s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.476998842 (304h52m55.348081877s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.477023177 (304h52m55.348106222s) [seg_pos={0 0}, upload=267329]: Segment reached optimal threshold of 13 pieces. Dec 18 08:21:55.477226615 (304h52m55.34830965s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477234159 (304h52m55.348317204s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477517129 (304h52m55.348600174s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477544811 (304h52m55.348627856s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477714985 (304h52m55.34879803s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477722289 (304h52m55.348805334s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477740153 (304h52m55.348823198s) [seg_pos={0 0}, upload=267329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:21:55.477862617 (304h52m55.348945662s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:21:55.477867526 (304h52m55.348950561s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:21:55.773001729 (304h52m55.644084784s) [upload=267329]: Done uploading segments... Dec 18 08:21:55.773021757 (304h52m55.644104802s) [upload=267329]: Upload finished. err: <nil> Duration: 6.181848988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.032898421s trace: https://jaeger.dev.storj.io/trace/162d3bebbccb0cd4 FAIL (2025-12-18 08:26:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 08:21:52.979634680 (304h52m52.850717715s) [upload=267329]: Starting upload Dec 18 08:21:52.979689916 (304h52m52.850772961s) [upload=267329]: Done starting upload Dec 18 08:21:52.979868376 (304h52m52.850951421s) [upload=267329]: Uploading segments... Dec 18 08:21:52.979915966 (304h52m52.850999001s) [upload=267329]: Got next segment. Inline: false Dec 18 08:21:52.979939922 (304h52m52.851022957s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:21:52.979943899 (304h52m52.851026944s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:21:53.227436064 (304h52m53.09851911s) [seg_pos={0 0}, upload=267329]: Begin upload segment... Dec 18 08:21:53.227511688 (304h52m53.098594723s) [seg_pos={0 0}, upload=267329]: Done begin upload segment. Dec 18 08:21:53.227584286 (304h52m53.098667321s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.229933821 (304h52m53.101016866s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.230418473 (304h52m53.101501508s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231688963 (304h52m53.102771998s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231930132 (304h52m53.103013167s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.232747668 (304h52m53.103830703s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.233027391 (304h52m53.104110426s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234000514 (304h52m53.105083549s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234145360 (304h52m53.105228395s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235434395 (304h52m53.10651744s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235584220 (304h52m53.106667255s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.236732076 (304h52m53.107815111s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.237519464 (304h52m53.108602499s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.239600678 (304h52m53.110683723s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.240042359 (304h52m53.111125404s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.242933985 (304h52m53.11401702s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:54.851541589 (304h52m54.722624634s) [upload=267329]: Next returned nil segment Dec 18 08:21:54.851554333 (304h52m54.722637378s) [upload=267329]: Waiting for error group managing segments... Dec 18 08:21:55.235427483 (304h52m55.106510538s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236089233 (304h52m55.107172278s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236667895 (304h52m55.10775094s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.252488160 (304h52m55.123571195s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255770450 (304h52m55.126853495s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255998223 (304h52m55.127081258s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.256221949 (304h52m55.127304985s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.258454531 (304h52m55.129537566s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.262083320 (304h52m55.133166355s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.302758765 (304h52m55.17384181s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.303381982 (304h52m55.174465027s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.437044621 (304h52m55.308127666s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.476998842 (304h52m55.348081877s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.477023177 (304h52m55.348106222s) [seg_pos={0 0}, upload=267329]: Segment reached optimal threshold of 13 pieces. Dec 18 08:21:55.477226615 (304h52m55.34830965s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477234159 (304h52m55.348317204s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477517129 (304h52m55.348600174s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477544811 (304h52m55.348627856s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477714985 (304h52m55.34879803s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477722289 (304h52m55.348805334s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477740153 (304h52m55.348823198s) [seg_pos={0 0}, upload=267329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:21:55.477862617 (304h52m55.348945662s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:21:55.477867526 (304h52m55.348950561s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:21:55.773001729 (304h52m55.644084784s) [upload=267329]: Done uploading segments... Dec 18 08:21:55.773021757 (304h52m55.644104802s) [upload=267329]: Upload finished. err: <nil> Duration: 6.032927837s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.678837328s trace: https://jaeger.dev.storj.io/trace/162d3bebbccb0cd4 FAIL (2025-12-18 08:26:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 08:21:52.979634680 (304h52m52.850717715s) [upload=267329]: Starting upload Dec 18 08:21:52.979689916 (304h52m52.850772961s) [upload=267329]: Done starting upload Dec 18 08:21:52.979868376 (304h52m52.850951421s) [upload=267329]: Uploading segments... Dec 18 08:21:52.979915966 (304h52m52.850999001s) [upload=267329]: Got next segment. Inline: false Dec 18 08:21:52.979939922 (304h52m52.851022957s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:21:52.979943899 (304h52m52.851026944s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:21:53.227436064 (304h52m53.09851911s) [seg_pos={0 0}, upload=267329]: Begin upload segment... Dec 18 08:21:53.227511688 (304h52m53.098594723s) [seg_pos={0 0}, upload=267329]: Done begin upload segment. Dec 18 08:21:53.227584286 (304h52m53.098667321s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.229933821 (304h52m53.101016866s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.230418473 (304h52m53.101501508s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231688963 (304h52m53.102771998s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231930132 (304h52m53.103013167s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.232747668 (304h52m53.103830703s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.233027391 (304h52m53.104110426s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234000514 (304h52m53.105083549s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234145360 (304h52m53.105228395s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235434395 (304h52m53.10651744s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235584220 (304h52m53.106667255s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.236732076 (304h52m53.107815111s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.237519464 (304h52m53.108602499s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.239600678 (304h52m53.110683723s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.240042359 (304h52m53.111125404s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.242933985 (304h52m53.11401702s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:54.851541589 (304h52m54.722624634s) [upload=267329]: Next returned nil segment Dec 18 08:21:54.851554333 (304h52m54.722637378s) [upload=267329]: Waiting for error group managing segments... Dec 18 08:21:55.235427483 (304h52m55.106510538s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236089233 (304h52m55.107172278s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236667895 (304h52m55.10775094s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.252488160 (304h52m55.123571195s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255770450 (304h52m55.126853495s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255998223 (304h52m55.127081258s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.256221949 (304h52m55.127304985s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.258454531 (304h52m55.129537566s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.262083320 (304h52m55.133166355s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.302758765 (304h52m55.17384181s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.303381982 (304h52m55.174465027s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.437044621 (304h52m55.308127666s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.476998842 (304h52m55.348081877s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.477023177 (304h52m55.348106222s) [seg_pos={0 0}, upload=267329]: Segment reached optimal threshold of 13 pieces. Dec 18 08:21:55.477226615 (304h52m55.34830965s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477234159 (304h52m55.348317204s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477517129 (304h52m55.348600174s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477544811 (304h52m55.348627856s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477714985 (304h52m55.34879803s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477722289 (304h52m55.348805334s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477740153 (304h52m55.348823198s) [seg_pos={0 0}, upload=267329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:21:55.477862617 (304h52m55.348945662s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:21:55.477867526 (304h52m55.348950561s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:21:55.773001729 (304h52m55.644084784s) [upload=267329]: Done uploading segments... Dec 18 08:21:55.773021757 (304h52m55.644104802s) [upload=267329]: Upload finished. err: <nil> Duration: 6.67887518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.051169993s trace: https://jaeger.dev.storj.io/trace/162d3bebbccb0cd4 FAIL (2025-12-18 08:25:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 08:21:52.979634680 (304h52m52.850717715s) [upload=267329]: Starting upload Dec 18 08:21:52.979689916 (304h52m52.850772961s) [upload=267329]: Done starting upload Dec 18 08:21:52.979868376 (304h52m52.850951421s) [upload=267329]: Uploading segments... Dec 18 08:21:52.979915966 (304h52m52.850999001s) [upload=267329]: Got next segment. Inline: false Dec 18 08:21:52.979939922 (304h52m52.851022957s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:21:52.979943899 (304h52m52.851026944s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:21:53.227436064 (304h52m53.09851911s) [seg_pos={0 0}, upload=267329]: Begin upload segment... Dec 18 08:21:53.227511688 (304h52m53.098594723s) [seg_pos={0 0}, upload=267329]: Done begin upload segment. Dec 18 08:21:53.227584286 (304h52m53.098667321s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.229933821 (304h52m53.101016866s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.230418473 (304h52m53.101501508s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231688963 (304h52m53.102771998s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231930132 (304h52m53.103013167s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.232747668 (304h52m53.103830703s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.233027391 (304h52m53.104110426s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234000514 (304h52m53.105083549s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234145360 (304h52m53.105228395s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235434395 (304h52m53.10651744s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235584220 (304h52m53.106667255s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.236732076 (304h52m53.107815111s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.237519464 (304h52m53.108602499s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.239600678 (304h52m53.110683723s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.240042359 (304h52m53.111125404s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.242933985 (304h52m53.11401702s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:54.851541589 (304h52m54.722624634s) [upload=267329]: Next returned nil segment Dec 18 08:21:54.851554333 (304h52m54.722637378s) [upload=267329]: Waiting for error group managing segments... Dec 18 08:21:55.235427483 (304h52m55.106510538s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236089233 (304h52m55.107172278s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236667895 (304h52m55.10775094s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.252488160 (304h52m55.123571195s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255770450 (304h52m55.126853495s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255998223 (304h52m55.127081258s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.256221949 (304h52m55.127304985s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.258454531 (304h52m55.129537566s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.262083320 (304h52m55.133166355s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.302758765 (304h52m55.17384181s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.303381982 (304h52m55.174465027s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.437044621 (304h52m55.308127666s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.476998842 (304h52m55.348081877s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.477023177 (304h52m55.348106222s) [seg_pos={0 0}, upload=267329]: Segment reached optimal threshold of 13 pieces. Dec 18 08:21:55.477226615 (304h52m55.34830965s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477234159 (304h52m55.348317204s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477517129 (304h52m55.348600174s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477544811 (304h52m55.348627856s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477714985 (304h52m55.34879803s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477722289 (304h52m55.348805334s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477740153 (304h52m55.348823198s) [seg_pos={0 0}, upload=267329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:21:55.477862617 (304h52m55.348945662s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:21:55.477867526 (304h52m55.348950561s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:21:55.773001729 (304h52m55.644084784s) [upload=267329]: Done uploading segments... Dec 18 08:21:55.773021757 (304h52m55.644104802s) [upload=267329]: Upload finished. err: <nil> Duration: 6.051199349s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.157903583s trace: https://jaeger.dev.storj.io/trace/162d3bebbccb0cd4 FAIL (2025-12-18 08:25:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 08:21:52.979634680 (304h52m52.850717715s) [upload=267329]: Starting upload Dec 18 08:21:52.979689916 (304h52m52.850772961s) [upload=267329]: Done starting upload Dec 18 08:21:52.979868376 (304h52m52.850951421s) [upload=267329]: Uploading segments... Dec 18 08:21:52.979915966 (304h52m52.850999001s) [upload=267329]: Got next segment. Inline: false Dec 18 08:21:52.979939922 (304h52m52.851022957s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:21:52.979943899 (304h52m52.851026944s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:21:53.227436064 (304h52m53.09851911s) [seg_pos={0 0}, upload=267329]: Begin upload segment... Dec 18 08:21:53.227511688 (304h52m53.098594723s) [seg_pos={0 0}, upload=267329]: Done begin upload segment. Dec 18 08:21:53.227584286 (304h52m53.098667321s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.229933821 (304h52m53.101016866s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.230418473 (304h52m53.101501508s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231688963 (304h52m53.102771998s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231930132 (304h52m53.103013167s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.232747668 (304h52m53.103830703s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.233027391 (304h52m53.104110426s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234000514 (304h52m53.105083549s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234145360 (304h52m53.105228395s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235434395 (304h52m53.10651744s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235584220 (304h52m53.106667255s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.236732076 (304h52m53.107815111s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.237519464 (304h52m53.108602499s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.239600678 (304h52m53.110683723s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.240042359 (304h52m53.111125404s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.242933985 (304h52m53.11401702s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:54.851541589 (304h52m54.722624634s) [upload=267329]: Next returned nil segment Dec 18 08:21:54.851554333 (304h52m54.722637378s) [upload=267329]: Waiting for error group managing segments... Dec 18 08:21:55.235427483 (304h52m55.106510538s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236089233 (304h52m55.107172278s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236667895 (304h52m55.10775094s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.252488160 (304h52m55.123571195s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255770450 (304h52m55.126853495s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255998223 (304h52m55.127081258s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.256221949 (304h52m55.127304985s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.258454531 (304h52m55.129537566s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.262083320 (304h52m55.133166355s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.302758765 (304h52m55.17384181s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.303381982 (304h52m55.174465027s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.437044621 (304h52m55.308127666s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.476998842 (304h52m55.348081877s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.477023177 (304h52m55.348106222s) [seg_pos={0 0}, upload=267329]: Segment reached optimal threshold of 13 pieces. Dec 18 08:21:55.477226615 (304h52m55.34830965s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477234159 (304h52m55.348317204s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477517129 (304h52m55.348600174s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477544811 (304h52m55.348627856s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477714985 (304h52m55.34879803s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477722289 (304h52m55.348805334s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477740153 (304h52m55.348823198s) [seg_pos={0 0}, upload=267329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:21:55.477862617 (304h52m55.348945662s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:21:55.477867526 (304h52m55.348950561s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:21:55.773001729 (304h52m55.644084784s) [upload=267329]: Done uploading segments... Dec 18 08:21:55.773021757 (304h52m55.644104802s) [upload=267329]: Upload finished. err: <nil> Duration: 6.157927878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.810692772s trace: https://jaeger.dev.storj.io/trace/162d3bebbccb0cd4 FAIL (2025-12-18 08:24:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 08:21:52.979634680 (304h52m52.850717715s) [upload=267329]: Starting upload Dec 18 08:21:52.979689916 (304h52m52.850772961s) [upload=267329]: Done starting upload Dec 18 08:21:52.979868376 (304h52m52.850951421s) [upload=267329]: Uploading segments... Dec 18 08:21:52.979915966 (304h52m52.850999001s) [upload=267329]: Got next segment. Inline: false Dec 18 08:21:52.979939922 (304h52m52.851022957s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:21:52.979943899 (304h52m52.851026944s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:21:53.227436064 (304h52m53.09851911s) [seg_pos={0 0}, upload=267329]: Begin upload segment... Dec 18 08:21:53.227511688 (304h52m53.098594723s) [seg_pos={0 0}, upload=267329]: Done begin upload segment. Dec 18 08:21:53.227584286 (304h52m53.098667321s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.229933821 (304h52m53.101016866s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.230418473 (304h52m53.101501508s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231688963 (304h52m53.102771998s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231930132 (304h52m53.103013167s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.232747668 (304h52m53.103830703s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.233027391 (304h52m53.104110426s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234000514 (304h52m53.105083549s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234145360 (304h52m53.105228395s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235434395 (304h52m53.10651744s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235584220 (304h52m53.106667255s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.236732076 (304h52m53.107815111s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.237519464 (304h52m53.108602499s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.239600678 (304h52m53.110683723s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.240042359 (304h52m53.111125404s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.242933985 (304h52m53.11401702s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:54.851541589 (304h52m54.722624634s) [upload=267329]: Next returned nil segment Dec 18 08:21:54.851554333 (304h52m54.722637378s) [upload=267329]: Waiting for error group managing segments... Dec 18 08:21:55.235427483 (304h52m55.106510538s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236089233 (304h52m55.107172278s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236667895 (304h52m55.10775094s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.252488160 (304h52m55.123571195s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255770450 (304h52m55.126853495s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255998223 (304h52m55.127081258s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.256221949 (304h52m55.127304985s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.258454531 (304h52m55.129537566s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.262083320 (304h52m55.133166355s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.302758765 (304h52m55.17384181s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.303381982 (304h52m55.174465027s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.437044621 (304h52m55.308127666s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.476998842 (304h52m55.348081877s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.477023177 (304h52m55.348106222s) [seg_pos={0 0}, upload=267329]: Segment reached optimal threshold of 13 pieces. Dec 18 08:21:55.477226615 (304h52m55.34830965s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477234159 (304h52m55.348317204s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477517129 (304h52m55.348600174s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477544811 (304h52m55.348627856s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477714985 (304h52m55.34879803s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477722289 (304h52m55.348805334s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477740153 (304h52m55.348823198s) [seg_pos={0 0}, upload=267329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:21:55.477862617 (304h52m55.348945662s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:21:55.477867526 (304h52m55.348950561s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:21:55.773001729 (304h52m55.644084784s) [upload=267329]: Done uploading segments... Dec 18 08:21:55.773021757 (304h52m55.644104802s) [upload=267329]: Upload finished. err: <nil> Duration: 5.810729631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.984333254s trace: https://jaeger.dev.storj.io/trace/162d3bebbccb0cd4 FAIL (2025-12-18 08:23:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 08:21:52.979634680 (304h52m52.850717715s) [upload=267329]: Starting upload Dec 18 08:21:52.979689916 (304h52m52.850772961s) [upload=267329]: Done starting upload Dec 18 08:21:52.979868376 (304h52m52.850951421s) [upload=267329]: Uploading segments... Dec 18 08:21:52.979915966 (304h52m52.850999001s) [upload=267329]: Got next segment. Inline: false Dec 18 08:21:52.979939922 (304h52m52.851022957s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:21:52.979943899 (304h52m52.851026944s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:21:53.227436064 (304h52m53.09851911s) [seg_pos={0 0}, upload=267329]: Begin upload segment... Dec 18 08:21:53.227511688 (304h52m53.098594723s) [seg_pos={0 0}, upload=267329]: Done begin upload segment. Dec 18 08:21:53.227584286 (304h52m53.098667321s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.229933821 (304h52m53.101016866s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.230418473 (304h52m53.101501508s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231688963 (304h52m53.102771998s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231930132 (304h52m53.103013167s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.232747668 (304h52m53.103830703s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.233027391 (304h52m53.104110426s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234000514 (304h52m53.105083549s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234145360 (304h52m53.105228395s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235434395 (304h52m53.10651744s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235584220 (304h52m53.106667255s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.236732076 (304h52m53.107815111s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.237519464 (304h52m53.108602499s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.239600678 (304h52m53.110683723s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.240042359 (304h52m53.111125404s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.242933985 (304h52m53.11401702s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:54.851541589 (304h52m54.722624634s) [upload=267329]: Next returned nil segment Dec 18 08:21:54.851554333 (304h52m54.722637378s) [upload=267329]: Waiting for error group managing segments... Dec 18 08:21:55.235427483 (304h52m55.106510538s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236089233 (304h52m55.107172278s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236667895 (304h52m55.10775094s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.252488160 (304h52m55.123571195s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255770450 (304h52m55.126853495s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255998223 (304h52m55.127081258s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.256221949 (304h52m55.127304985s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.258454531 (304h52m55.129537566s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.262083320 (304h52m55.133166355s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.302758765 (304h52m55.17384181s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.303381982 (304h52m55.174465027s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.437044621 (304h52m55.308127666s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.476998842 (304h52m55.348081877s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.477023177 (304h52m55.348106222s) [seg_pos={0 0}, upload=267329]: Segment reached optimal threshold of 13 pieces. Dec 18 08:21:55.477226615 (304h52m55.34830965s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477234159 (304h52m55.348317204s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477517129 (304h52m55.348600174s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477544811 (304h52m55.348627856s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477714985 (304h52m55.34879803s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477722289 (304h52m55.348805334s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477740153 (304h52m55.348823198s) [seg_pos={0 0}, upload=267329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:21:55.477862617 (304h52m55.348945662s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:21:55.477867526 (304h52m55.348950561s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:21:55.773001729 (304h52m55.644084784s) [upload=267329]: Done uploading segments... Dec 18 08:21:55.773021757 (304h52m55.644104802s) [upload=267329]: Upload finished. err: <nil> Duration: 5.984350938s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.99065173s trace: https://jaeger.dev.storj.io/trace/162d3bebbccb0cd4 FAIL (2025-12-18 08:22:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 08:21:52.979634680 (304h52m52.850717715s) [upload=267329]: Starting upload Dec 18 08:21:52.979689916 (304h52m52.850772961s) [upload=267329]: Done starting upload Dec 18 08:21:52.979868376 (304h52m52.850951421s) [upload=267329]: Uploading segments... Dec 18 08:21:52.979915966 (304h52m52.850999001s) [upload=267329]: Got next segment. Inline: false Dec 18 08:21:52.979939922 (304h52m52.851022957s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:21:52.979943899 (304h52m52.851026944s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:21:53.227436064 (304h52m53.09851911s) [seg_pos={0 0}, upload=267329]: Begin upload segment... Dec 18 08:21:53.227511688 (304h52m53.098594723s) [seg_pos={0 0}, upload=267329]: Done begin upload segment. Dec 18 08:21:53.227584286 (304h52m53.098667321s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.229933821 (304h52m53.101016866s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.230418473 (304h52m53.101501508s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231688963 (304h52m53.102771998s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231930132 (304h52m53.103013167s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.232747668 (304h52m53.103830703s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.233027391 (304h52m53.104110426s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234000514 (304h52m53.105083549s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234145360 (304h52m53.105228395s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235434395 (304h52m53.10651744s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235584220 (304h52m53.106667255s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.236732076 (304h52m53.107815111s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.237519464 (304h52m53.108602499s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.239600678 (304h52m53.110683723s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.240042359 (304h52m53.111125404s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.242933985 (304h52m53.11401702s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:54.851541589 (304h52m54.722624634s) [upload=267329]: Next returned nil segment Dec 18 08:21:54.851554333 (304h52m54.722637378s) [upload=267329]: Waiting for error group managing segments... Dec 18 08:21:55.235427483 (304h52m55.106510538s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236089233 (304h52m55.107172278s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236667895 (304h52m55.10775094s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.252488160 (304h52m55.123571195s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255770450 (304h52m55.126853495s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255998223 (304h52m55.127081258s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.256221949 (304h52m55.127304985s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.258454531 (304h52m55.129537566s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.262083320 (304h52m55.133166355s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.302758765 (304h52m55.17384181s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.303381982 (304h52m55.174465027s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.437044621 (304h52m55.308127666s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.476998842 (304h52m55.348081877s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.477023177 (304h52m55.348106222s) [seg_pos={0 0}, upload=267329]: Segment reached optimal threshold of 13 pieces. Dec 18 08:21:55.477226615 (304h52m55.34830965s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477234159 (304h52m55.348317204s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477517129 (304h52m55.348600174s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477544811 (304h52m55.348627856s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477714985 (304h52m55.34879803s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477722289 (304h52m55.348805334s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477740153 (304h52m55.348823198s) [seg_pos={0 0}, upload=267329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:21:55.477862617 (304h52m55.348945662s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:21:55.477867526 (304h52m55.348950561s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:21:55.773001729 (304h52m55.644084784s) [upload=267329]: Done uploading segments... Dec 18 08:21:55.773021757 (304h52m55.644104802s) [upload=267329]: Upload finished. err: <nil> Duration: 5.990667039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002400425s trace: https://jaeger.dev.storj.io/trace/162d3bebbccb0cd4 FAIL (2025-12-18 08:22:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 08:21:52.979634680 (304h52m52.850717715s) [upload=267329]: Starting upload Dec 18 08:21:52.979689916 (304h52m52.850772961s) [upload=267329]: Done starting upload Dec 18 08:21:52.979868376 (304h52m52.850951421s) [upload=267329]: Uploading segments... Dec 18 08:21:52.979915966 (304h52m52.850999001s) [upload=267329]: Got next segment. Inline: false Dec 18 08:21:52.979939922 (304h52m52.851022957s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 08:21:52.979943899 (304h52m52.851026944s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 08:21:53.227436064 (304h52m53.09851911s) [seg_pos={0 0}, upload=267329]: Begin upload segment... Dec 18 08:21:53.227511688 (304h52m53.098594723s) [seg_pos={0 0}, upload=267329]: Done begin upload segment. Dec 18 08:21:53.227584286 (304h52m53.098667321s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.229933821 (304h52m53.101016866s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.230418473 (304h52m53.101501508s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231688963 (304h52m53.102771998s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.231930132 (304h52m53.103013167s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.232747668 (304h52m53.103830703s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.233027391 (304h52m53.104110426s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234000514 (304h52m53.105083549s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.234145360 (304h52m53.105228395s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235434395 (304h52m53.10651744s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.235584220 (304h52m53.106667255s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.236732076 (304h52m53.107815111s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.237519464 (304h52m53.108602499s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.239600678 (304h52m53.110683723s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.240042359 (304h52m53.111125404s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:53.242933985 (304h52m53.11401702s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Uploading piece... Dec 18 08:21:54.851541589 (304h52m54.722624634s) [upload=267329]: Next returned nil segment Dec 18 08:21:54.851554333 (304h52m54.722637378s) [upload=267329]: Waiting for error group managing segments... Dec 18 08:21:55.235427483 (304h52m55.106510538s) [noise=true , address=145.79.227.36:4518 , piece_id=RRXA5NBL2IN5L64DGHHSM3KEUQLOVDNIEG5WRH3HHBLQ7XEYCLGA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236089233 (304h52m55.107172278s) [noise=true , address=145.79.227.50:4375 , piece_id=UVLRXB65SUN6D66KPUN7LFQDIQ3OLEMVN7CUH2INCJZBZSDJFH7Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.236667895 (304h52m55.10775094s) [noise=true , address=145.79.227.32:4835 , piece_id=SOBDSVQXMTAAXCYSKCARDUBRPIMGHOMW4WCMDPI44SDECQMHWCSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.252488160 (304h52m55.123571195s) [noise=true , address=145.79.227.35:4482 , piece_id=JCYHIYWSA4KILU2T4OUGYF2XI2DIU6BMYSLICEY6VB2K3B462WCQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255770450 (304h52m55.126853495s) [noise=true , address=145.79.227.38:4977 , piece_id=XLXZSXY5XDK7WE6KNVLFOWDD7RTMPNI2L7NMGBZM7CZV4LLO32QA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.255998223 (304h52m55.127081258s) [noise=true , address=145.79.227.27:4689 , piece_id=36POCJJ5DT7AODY3MWEJSZ3HVNVJ46YWVJXW5EP6CQ23IWHUGFSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.256221949 (304h52m55.127304985s) [noise=true , address=145.79.227.49:4338 , piece_id=AGVLWHJCIRPOYMFSVKGYYAF2DYBIOQCTUGKQCNK7N4QTYPCYMBDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.258454531 (304h52m55.129537566s) [noise=true , address=145.79.227.9:4366 , piece_id=FXDT2AW24LTMMG3EMDPPHAHZ7NASIURY723NYK4VLY6OQEKJDWDA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.262083320 (304h52m55.133166355s) [noise=true , address=145.79.227.45:4229 , piece_id=4JJREQISHYNLJAKQQF6UP6AFGXOOV5HVONESCEP5XSMKVOU3CIFA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.302758765 (304h52m55.17384181s) [noise=true , address=145.79.227.20:4439 , piece_id=XL64MF66HMLQNAIUFGUPRJFU3M6AJVEVQT46R2NOCPYZ7LRA2PIQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.303381982 (304h52m55.174465027s) [noise=true , address=145.79.227.29:4941 , piece_id=IW74NRFNI4GDLIY6PLQEKKYGMU4KTKLV3Y4ZLXLOPP3OKZIA5DYQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.437044621 (304h52m55.308127666s) [noise=true , address=77.90.31.12:4590 , piece_id=3HUL6H5JQ24647U5HSM2KRVKE73NIJJG4P5ECJMSMCDNI4FRTZZQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.476998842 (304h52m55.348081877s) [noise=true , address=77.90.31.13:4627 , piece_id=X6ZWAKLC5KTD7U6BZZVIMI7W5EPSZUMVZ3BSJBNJQUZJ23SQAV2Q, seg_pos={0 0}, upload=267329]: Done uploading piece. err: <nil> Dec 18 08:21:55.477023177 (304h52m55.348106222s) [seg_pos={0 0}, upload=267329]: Segment reached optimal threshold of 13 pieces. Dec 18 08:21:55.477226615 (304h52m55.34830965s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477234159 (304h52m55.348317204s) [noise=true , address=45.91.116.8:4047 , piece_id=3SSC24RQ6Y5YDVJ3QO4CKGSFPKBXSFRGUDKOWTJ2TXL5XGJFXSSA, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477517129 (304h52m55.348600174s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477544811 (304h52m55.348627856s) [noise=true , address=45.91.116.9:4264 , piece_id=2FWM63MJN4AYYC353NJYDDBQSXPDZK4CV343ND33ZQ3VYFIJVUOQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477714985 (304h52m55.34879803s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 18 08:21:55.477722289 (304h52m55.348805334s) [noise=true , address=77.90.31.15:4700 , piece_id=JUCTTSBXQR4SKUMYA6CEYD7HPECGJ5EFB7JUS3TTFIOUA4XA2UHQ, seg_pos={0 0}, upload=267329]: Long tail cancellation due to optimal threshold Dec 18 08:21:55.477740153 (304h52m55.348823198s) [seg_pos={0 0}, upload=267329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 08:21:55.477862617 (304h52m55.348945662s) [upload=267329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 08:21:55.477867526 (304h52m55.348950561s) [upload=267329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 08:21:55.773001729 (304h52m55.644084784s) [upload=267329]: Done uploading segments... Dec 18 08:21:55.773021757 (304h52m55.644104802s) [upload=267329]: Upload finished. err: <nil> Duration: 30.002417307s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.766125474s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:50:58 UTC) orbiter warning: context deadline exceeded Duration: 5.766143189s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.079398529s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:50:22 UTC) orbiter warning: context deadline exceeded Duration: 6.079424428s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.126338836s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:50:01 UTC) orbiter warning: context deadline exceeded Duration: 6.126369474s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.132320033s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:49:22 UTC) orbiter warning: context deadline exceeded Duration: 6.132345891s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.751956802s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:48:49 UTC) orbiter warning: context deadline exceeded Duration: 5.751981328s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.206423716s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:48:20 UTC) orbiter warning: context deadline exceeded Duration: 6.206441881s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.628726362s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:47:28 UTC) orbiter warning: context deadline exceeded Duration: 5.62875219s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.873772076s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:46:43 UTC) orbiter warning: context deadline exceeded Duration: 5.873797934s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.8769195s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:46:17 UTC) orbiter warning: context deadline exceeded Duration: 5.876950099s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.630050946s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:45:11 UTC) orbiter warning: context deadline exceeded Duration: 5.630082636s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.107795693s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:44:41 UTC) orbiter warning: context deadline exceeded Duration: 6.10782011s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.406176977s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:43:44 UTC) orbiter warning: context deadline exceeded Duration: 6.40621515s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.126124461s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:42:49 UTC) orbiter warning: context deadline exceeded Duration: 6.126151983s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.980941963s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:42:28 UTC) orbiter warning: context deadline exceeded Duration: 5.980969847s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.834484064s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:41:50 UTC) orbiter warning: context deadline exceeded Duration: 5.834509554s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.966772602s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:41:35 UTC) orbiter warning: context deadline exceeded Duration: 5.966802117s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.807302703s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:41:15 UTC) orbiter warning: context deadline exceeded Duration: 5.807322902s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.341934794s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:40:26 UTC) orbiter warning: context deadline exceeded Duration: 6.341962487s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.780147025s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:39:35 UTC) orbiter warning: context deadline exceeded Duration: 5.780177633s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.485688417s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:38:58 UTC) orbiter warning: context deadline exceeded Duration: 6.485703946s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.759143541s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:38:02 UTC) orbiter warning: context deadline exceeded Duration: 5.75917464s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.668008248s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:37:32 UTC) orbiter warning: context deadline exceeded Duration: 5.668033706s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.996130467s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:36:59 UTC) orbiter warning: context deadline exceeded Duration: 5.996165353s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.16789248s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:36:08 UTC) orbiter warning: context deadline exceeded Duration: 6.167922166s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.676387708s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:35:01 UTC) orbiter warning: context deadline exceeded Duration: 5.676416292s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.897822773s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:33:47 UTC) orbiter warning: context deadline exceeded Duration: 5.897903707s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.842939516s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:33:03 UTC) orbiter warning: context deadline exceeded Duration: 5.842973641s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.985421944s trace: https://jaeger.dev.storj.io/trace/2a0e1749283690b2 FAIL (2025-12-18 07:32:10 UTC) orbiter warning: context deadline exceeded Duration: 5.985453303s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)