Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X..X..X....X.X.X.XX..X..X..X...X...XX.......X.X...X.....X.] duration: 6.091574502s trace: https://jaeger.dev.storj.io/trace/22695cdd2ef11057 FAIL (2025-12-28 18:53:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: JXVBOXOQYIGUOLYKHYPIFHCY5VHVOMHNZX3UA5LVMLB5MWZKQVPA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: OUAC3GM72TBC2PMIAJWHSCOV3XI3FABSYPKIVROGOJP5EK5S3JMQ) context deadline exceeded ======LOGS====== 52:29.400503831 (555h23m29.271586866s) [upload=486909]: Starting upload Dec 28 18:52:29.400632736 (555h23m29.271715801s) [upload=486909]: Done starting upload Dec 28 18:52:29.400710986 (555h23m29.271794021s) [upload=486909]: Uploading segments... Dec 28 18:52:29.401095638 (555h23m29.272178683s) [upload=486909]: Got next segment. Inline: false Dec 28 18:52:29.401139872 (555h23m29.272222907s) [upload=486909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:52:29.401144721 (555h23m29.272227756s) [upload=486909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:52:29.647004279 (555h23m29.518087324s) [seg_pos={0 0}, upload=486909]: Begin upload segment... Dec 28 18:52:29.647059956 (555h23m29.518142991s) [seg_pos={0 0}, upload=486909]: Done begin upload segment. Dec 28 18:52:29.647111745 (555h23m29.51819479s) [noise=true , address=45.91.116.10:4083 , piece_id=L56QQA56NZVLOTHCH34YPTJC4I3V6N5WK6XC4TGEDGLEQ6BJNCLA, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.647131572 (555h23m29.518214617s) [noise=true , address=145.79.227.5:4221 , piece_id=CXOODW7DRRYNGKVHIM5OTHLGMKYMAYJDP2GLJL6WDOUPMGILOE7A, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.647212135 (555h23m29.51829518s) [noise=true , address=145.79.227.39:4014 , piece_id=XOLF4IEV2VS2TEXFHQAQIZ67OBDHAUF2IYXF7IXGFRKOEOCI5LEQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.647566521 (555h23m29.518649556s) [noise=true , address=145.79.227.11:4730 , piece_id=NRJ72HWUBHQWN3KJHX4JXG5YJZSDIPP6GDKBF5P5PS6MRAAV35OQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.647887953 (555h23m29.518970988s) [noise=true , address=145.79.227.33:4871 , piece_id=IVTCFLP42MJHZ3GUJVJCUUBZZGWDSIOPW6BZRZKYPXUFNONFPQPQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.648171673 (555h23m29.519254708s) [noise=true , address=45.91.116.9:4262 , piece_id=JXVBOXOQYIGUOLYKHYPIFHCY5VHVOMHNZX3UA5LVMLB5MWZKQVPA, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.648455693 (555h23m29.519538728s) [noise=true , address=145.79.227.2:4114 , piece_id=UFACKEVIEOOPAZYNQTXXCUFYP4MAENOZ47W3B5H45S5TGF4F4FVA, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.648811431 (555h23m29.519894476s) [noise=true , address=45.91.116.13:4154 , piece_id=OUAC3GM72TBC2PMIAJWHSCOV3XI3FABSYPKIVROGOJP5EK5S3JMQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.649124306 (555h23m29.520207341s) [noise=true , address=145.79.227.17:4972 , piece_id=I3ITEOY6FV7U7O4R32EFXN7RGAKPZYWW7APMDV64NT37DL5FAHJQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.649446200 (555h23m29.520529245s) [noise=true , address=145.79.227.29:4942 , piece_id=3TW6OEBS67GC4M2PZUHMI5ZMCKEVDQZJVKA3A3XUA3ZLMX3WDAMA, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.650616698 (555h23m29.521699733s) [noise=true , address=145.79.227.21:4475 , piece_id=DVBPKDDQ4DKGTDHJ7FYDISHUDVYAM535DNI3ZIWK3HHVSMHJKTFQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.651717705 (555h23m29.52280074s) [noise=true , address=45.91.116.12:4227 , piece_id=AR4544HO6IQAJLQJZSXBW32FBKQWQHYH26EXE2UJAAQNBV7WZJOQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.652821397 (555h23m29.523904432s) [noise=true , address=145.79.227.48:4302 , piece_id=CYY54F3IN2QC4WTAUKVZMEQEL3LN4JI6JSNMPQAMZNKVX3BKGG6Q, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.653215477 (555h23m29.524298522s) [noise=true , address=145.79.227.7:4295 , piece_id=XBUHEYDODJ3BMRK7XHJIJ54GX767IVWZAGGXUZWSLTMU6WVIKG3Q, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.653901834 (555h23m29.524984869s) [noise=true , address=145.79.227.51:4410 , piece_id=YX2JLNYRJVCTWPOLP3FCRMBCFLNKGOYIAR23SZANH3GVY5Q4WYKQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.656652171 (555h23m29.527735206s) [noise=true , address=145.79.227.9:4366 , piece_id=VZLGOUIQR7PJYHNM3JQTU4UVK3ZIEXVNVL4W4X6ULB4GQHQQWESQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:31.357473770 (555h23m31.228556815s) [upload=486909]: Next returned nil segment Dec 28 18:52:31.357491684 (555h23m31.228574729s) [upload=486909]: Waiting for error group managing segments... Dec 28 18:52:31.642370122 (555h23m31.513453157s) [noise=true , address=145.79.227.39:4014 , piece_id=XOLF4IEV2VS2TEXFHQAQIZ67OBDHAUF2IYXF7IXGFRKOEOCI5LEQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.647439125 (555h23m31.51852216s) [noise=true , address=145.79.227.2:4114 , piece_id=UFACKEVIEOOPAZYNQTXXCUFYP4MAENOZ47W3B5H45S5TGF4F4FVA, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.648838118 (555h23m31.519921153s) [noise=true , address=145.79.227.7:4295 , piece_id=XBUHEYDODJ3BMRK7XHJIJ54GX767IVWZAGGXUZWSLTMU6WVIKG3Q, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.661246886 (555h23m31.532329931s) [noise=true , address=145.79.227.17:4972 , piece_id=I3ITEOY6FV7U7O4R32EFXN7RGAKPZYWW7APMDV64NT37DL5FAHJQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.666044702 (555h23m31.537127747s) [noise=true , address=145.79.227.5:4221 , piece_id=CXOODW7DRRYNGKVHIM5OTHLGMKYMAYJDP2GLJL6WDOUPMGILOE7A, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.671209568 (555h23m31.542292613s) [noise=true , address=145.79.227.51:4410 , piece_id=YX2JLNYRJVCTWPOLP3FCRMBCFLNKGOYIAR23SZANH3GVY5Q4WYKQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.679573754 (555h23m31.550656799s) [noise=true , address=145.79.227.9:4366 , piece_id=VZLGOUIQR7PJYHNM3JQTU4UVK3ZIEXVNVL4W4X6ULB4GQHQQWESQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.684082660 (555h23m31.555165705s) [noise=true , address=145.79.227.21:4475 , piece_id=DVBPKDDQ4DKGTDHJ7FYDISHUDVYAM535DNI3ZIWK3HHVSMHJKTFQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.891810961 (555h23m31.762894006s) [noise=true , address=45.91.116.9:4262 , piece_id=JXVBOXOQYIGUOLYKHYPIFHCY5VHVOMHNZX3UA5LVMLB5MWZKQVPA, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.921102737 (555h23m31.792185782s) [noise=true , address=45.91.116.13:4154 , piece_id=OUAC3GM72TBC2PMIAJWHSCOV3XI3FABSYPKIVROGOJP5EK5S3JMQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:32.010882344 (555h23m31.881965389s) [noise=true , address=45.91.116.12:4227 , piece_id=AR4544HO6IQAJLQJZSXBW32FBKQWQHYH26EXE2UJAAQNBV7WZJOQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:32.782255702 (555h23m32.653338737s) [noise=true , address=45.91.116.10:4083 , piece_id=L56QQA56NZVLOTHCH34YPTJC4I3V6N5WK6XC4TGEDGLEQ6BJNCLA, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:33.568116466 (555h23m33.439199501s) [noise=true , address=145.79.227.29:4942 , piece_id=3TW6OEBS67GC4M2PZUHMI5ZMCKEVDQZJVKA3A3XUA3ZLMX3WDAMA, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:33.568203662 (555h23m33.439286718s) [seg_pos={0 0}, upload=486909]: Segment reached optimal threshold of 13 pieces. Dec 28 18:52:33.568441996 (555h23m33.439525042s) [noise=true , address=145.79.227.33:4871 , piece_id=IVTCFLP42MJHZ3GUJVJCUUBZZGWDSIOPW6BZRZKYPXUFNONFPQPQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 28 18:52:33.568450603 (555h23m33.439533648s) [noise=true , address=145.79.227.33:4871 , piece_id=IVTCFLP42MJHZ3GUJVJCUUBZZGWDSIOPW6BZRZKYPXUFNONFPQPQ, seg_pos={0 0}, upload=486909]: Long tail cancellation due to optimal threshold Dec 28 18:52:33.568569669 (555h23m33.439652724s) [noise=true , address=145.79.227.48:4302 , piece_id=CYY54F3IN2QC4WTAUKVZMEQEL3LN4JI6JSNMPQAMZNKVX3BKGG6Q, seg_pos={0 0}, upload=486909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: expected piece hash; context canceled; EOF Dec 28 18:52:33.568577244 (555h23m33.439660279s) [noise=true , address=145.79.227.48:4302 , piece_id=CYY54F3IN2QC4WTAUKVZMEQEL3LN4JI6JSNMPQAMZNKVX3BKGG6Q, seg_pos={0 0}, upload=486909]: Long tail cancellation due to optimal threshold Dec 28 18:52:33.568623722 (555h23m33.439706757s) [noise=true , address=145.79.227.11:4730 , piece_id=NRJ72HWUBHQWN3KJHX4JXG5YJZSDIPP6GDKBF5P5PS6MRAAV35OQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:43922->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:43922->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43922->145.79.227.11:4730: use of closed network connection Dec 28 18:52:33.568631687 (555h23m33.439714722s) [noise=true , address=145.79.227.11:4730 , piece_id=NRJ72HWUBHQWN3KJHX4JXG5YJZSDIPP6GDKBF5P5PS6MRAAV35OQ, seg_pos={0 0}, upload=486909]: Long tail cancellation due to optimal threshold Dec 28 18:52:33.568648169 (555h23m33.439731204s) [seg_pos={0 0}, upload=486909]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:52:33.568743710 (555h23m33.439826745s) [upload=486909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:52:33.568749612 (555h23m33.439832647s) [upload=486909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:52:33.854028773 (555h23m33.725111808s) [upload=486909]: Done uploading segments... Dec 28 18:52:33.854047769 (555h23m33.725130814s) [upload=486909]: Upload finished. err: <nil> Duration: 6.091589861s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X..X..X....X.X.X.XX..X..X..X...X...XX.......X.X...X.....X] duration: 30.003142462s trace: https://jaeger.dev.storj.io/trace/22695cdd2ef11057 FAIL (2025-12-28 18:52:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: JXVBOXOQYIGUOLYKHYPIFHCY5VHVOMHNZX3UA5LVMLB5MWZKQVPA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: OUAC3GM72TBC2PMIAJWHSCOV3XI3FABSYPKIVROGOJP5EK5S3JMQ) context deadline exceeded ======LOGS====== 52:29.400503831 (555h23m29.271586866s) [upload=486909]: Starting upload Dec 28 18:52:29.400632736 (555h23m29.271715801s) [upload=486909]: Done starting upload Dec 28 18:52:29.400710986 (555h23m29.271794021s) [upload=486909]: Uploading segments... Dec 28 18:52:29.401095638 (555h23m29.272178683s) [upload=486909]: Got next segment. Inline: false Dec 28 18:52:29.401139872 (555h23m29.272222907s) [upload=486909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:52:29.401144721 (555h23m29.272227756s) [upload=486909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:52:29.647004279 (555h23m29.518087324s) [seg_pos={0 0}, upload=486909]: Begin upload segment... Dec 28 18:52:29.647059956 (555h23m29.518142991s) [seg_pos={0 0}, upload=486909]: Done begin upload segment. Dec 28 18:52:29.647111745 (555h23m29.51819479s) [noise=true , address=45.91.116.10:4083 , piece_id=L56QQA56NZVLOTHCH34YPTJC4I3V6N5WK6XC4TGEDGLEQ6BJNCLA, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.647131572 (555h23m29.518214617s) [noise=true , address=145.79.227.5:4221 , piece_id=CXOODW7DRRYNGKVHIM5OTHLGMKYMAYJDP2GLJL6WDOUPMGILOE7A, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.647212135 (555h23m29.51829518s) [noise=true , address=145.79.227.39:4014 , piece_id=XOLF4IEV2VS2TEXFHQAQIZ67OBDHAUF2IYXF7IXGFRKOEOCI5LEQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.647566521 (555h23m29.518649556s) [noise=true , address=145.79.227.11:4730 , piece_id=NRJ72HWUBHQWN3KJHX4JXG5YJZSDIPP6GDKBF5P5PS6MRAAV35OQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.647887953 (555h23m29.518970988s) [noise=true , address=145.79.227.33:4871 , piece_id=IVTCFLP42MJHZ3GUJVJCUUBZZGWDSIOPW6BZRZKYPXUFNONFPQPQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.648171673 (555h23m29.519254708s) [noise=true , address=45.91.116.9:4262 , piece_id=JXVBOXOQYIGUOLYKHYPIFHCY5VHVOMHNZX3UA5LVMLB5MWZKQVPA, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.648455693 (555h23m29.519538728s) [noise=true , address=145.79.227.2:4114 , piece_id=UFACKEVIEOOPAZYNQTXXCUFYP4MAENOZ47W3B5H45S5TGF4F4FVA, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.648811431 (555h23m29.519894476s) [noise=true , address=45.91.116.13:4154 , piece_id=OUAC3GM72TBC2PMIAJWHSCOV3XI3FABSYPKIVROGOJP5EK5S3JMQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.649124306 (555h23m29.520207341s) [noise=true , address=145.79.227.17:4972 , piece_id=I3ITEOY6FV7U7O4R32EFXN7RGAKPZYWW7APMDV64NT37DL5FAHJQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.649446200 (555h23m29.520529245s) [noise=true , address=145.79.227.29:4942 , piece_id=3TW6OEBS67GC4M2PZUHMI5ZMCKEVDQZJVKA3A3XUA3ZLMX3WDAMA, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.650616698 (555h23m29.521699733s) [noise=true , address=145.79.227.21:4475 , piece_id=DVBPKDDQ4DKGTDHJ7FYDISHUDVYAM535DNI3ZIWK3HHVSMHJKTFQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.651717705 (555h23m29.52280074s) [noise=true , address=45.91.116.12:4227 , piece_id=AR4544HO6IQAJLQJZSXBW32FBKQWQHYH26EXE2UJAAQNBV7WZJOQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.652821397 (555h23m29.523904432s) [noise=true , address=145.79.227.48:4302 , piece_id=CYY54F3IN2QC4WTAUKVZMEQEL3LN4JI6JSNMPQAMZNKVX3BKGG6Q, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.653215477 (555h23m29.524298522s) [noise=true , address=145.79.227.7:4295 , piece_id=XBUHEYDODJ3BMRK7XHJIJ54GX767IVWZAGGXUZWSLTMU6WVIKG3Q, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.653901834 (555h23m29.524984869s) [noise=true , address=145.79.227.51:4410 , piece_id=YX2JLNYRJVCTWPOLP3FCRMBCFLNKGOYIAR23SZANH3GVY5Q4WYKQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:29.656652171 (555h23m29.527735206s) [noise=true , address=145.79.227.9:4366 , piece_id=VZLGOUIQR7PJYHNM3JQTU4UVK3ZIEXVNVL4W4X6ULB4GQHQQWESQ, seg_pos={0 0}, upload=486909]: Uploading piece... Dec 28 18:52:31.357473770 (555h23m31.228556815s) [upload=486909]: Next returned nil segment Dec 28 18:52:31.357491684 (555h23m31.228574729s) [upload=486909]: Waiting for error group managing segments... Dec 28 18:52:31.642370122 (555h23m31.513453157s) [noise=true , address=145.79.227.39:4014 , piece_id=XOLF4IEV2VS2TEXFHQAQIZ67OBDHAUF2IYXF7IXGFRKOEOCI5LEQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.647439125 (555h23m31.51852216s) [noise=true , address=145.79.227.2:4114 , piece_id=UFACKEVIEOOPAZYNQTXXCUFYP4MAENOZ47W3B5H45S5TGF4F4FVA, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.648838118 (555h23m31.519921153s) [noise=true , address=145.79.227.7:4295 , piece_id=XBUHEYDODJ3BMRK7XHJIJ54GX767IVWZAGGXUZWSLTMU6WVIKG3Q, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.661246886 (555h23m31.532329931s) [noise=true , address=145.79.227.17:4972 , piece_id=I3ITEOY6FV7U7O4R32EFXN7RGAKPZYWW7APMDV64NT37DL5FAHJQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.666044702 (555h23m31.537127747s) [noise=true , address=145.79.227.5:4221 , piece_id=CXOODW7DRRYNGKVHIM5OTHLGMKYMAYJDP2GLJL6WDOUPMGILOE7A, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.671209568 (555h23m31.542292613s) [noise=true , address=145.79.227.51:4410 , piece_id=YX2JLNYRJVCTWPOLP3FCRMBCFLNKGOYIAR23SZANH3GVY5Q4WYKQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.679573754 (555h23m31.550656799s) [noise=true , address=145.79.227.9:4366 , piece_id=VZLGOUIQR7PJYHNM3JQTU4UVK3ZIEXVNVL4W4X6ULB4GQHQQWESQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.684082660 (555h23m31.555165705s) [noise=true , address=145.79.227.21:4475 , piece_id=DVBPKDDQ4DKGTDHJ7FYDISHUDVYAM535DNI3ZIWK3HHVSMHJKTFQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.891810961 (555h23m31.762894006s) [noise=true , address=45.91.116.9:4262 , piece_id=JXVBOXOQYIGUOLYKHYPIFHCY5VHVOMHNZX3UA5LVMLB5MWZKQVPA, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:31.921102737 (555h23m31.792185782s) [noise=true , address=45.91.116.13:4154 , piece_id=OUAC3GM72TBC2PMIAJWHSCOV3XI3FABSYPKIVROGOJP5EK5S3JMQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:32.010882344 (555h23m31.881965389s) [noise=true , address=45.91.116.12:4227 , piece_id=AR4544HO6IQAJLQJZSXBW32FBKQWQHYH26EXE2UJAAQNBV7WZJOQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:32.782255702 (555h23m32.653338737s) [noise=true , address=45.91.116.10:4083 , piece_id=L56QQA56NZVLOTHCH34YPTJC4I3V6N5WK6XC4TGEDGLEQ6BJNCLA, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:33.568116466 (555h23m33.439199501s) [noise=true , address=145.79.227.29:4942 , piece_id=3TW6OEBS67GC4M2PZUHMI5ZMCKEVDQZJVKA3A3XUA3ZLMX3WDAMA, seg_pos={0 0}, upload=486909]: Done uploading piece. err: <nil> Dec 28 18:52:33.568203662 (555h23m33.439286718s) [seg_pos={0 0}, upload=486909]: Segment reached optimal threshold of 13 pieces. Dec 28 18:52:33.568441996 (555h23m33.439525042s) [noise=true , address=145.79.227.33:4871 , piece_id=IVTCFLP42MJHZ3GUJVJCUUBZZGWDSIOPW6BZRZKYPXUFNONFPQPQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 28 18:52:33.568450603 (555h23m33.439533648s) [noise=true , address=145.79.227.33:4871 , piece_id=IVTCFLP42MJHZ3GUJVJCUUBZZGWDSIOPW6BZRZKYPXUFNONFPQPQ, seg_pos={0 0}, upload=486909]: Long tail cancellation due to optimal threshold Dec 28 18:52:33.568569669 (555h23m33.439652724s) [noise=true , address=145.79.227.48:4302 , piece_id=CYY54F3IN2QC4WTAUKVZMEQEL3LN4JI6JSNMPQAMZNKVX3BKGG6Q, seg_pos={0 0}, upload=486909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: expected piece hash; context canceled; EOF Dec 28 18:52:33.568577244 (555h23m33.439660279s) [noise=true , address=145.79.227.48:4302 , piece_id=CYY54F3IN2QC4WTAUKVZMEQEL3LN4JI6JSNMPQAMZNKVX3BKGG6Q, seg_pos={0 0}, upload=486909]: Long tail cancellation due to optimal threshold Dec 28 18:52:33.568623722 (555h23m33.439706757s) [noise=true , address=145.79.227.11:4730 , piece_id=NRJ72HWUBHQWN3KJHX4JXG5YJZSDIPP6GDKBF5P5PS6MRAAV35OQ, seg_pos={0 0}, upload=486909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:43922->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:43922->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43922->145.79.227.11:4730: use of closed network connection Dec 28 18:52:33.568631687 (555h23m33.439714722s) [noise=true , address=145.79.227.11:4730 , piece_id=NRJ72HWUBHQWN3KJHX4JXG5YJZSDIPP6GDKBF5P5PS6MRAAV35OQ, seg_pos={0 0}, upload=486909]: Long tail cancellation due to optimal threshold Dec 28 18:52:33.568648169 (555h23m33.439731204s) [seg_pos={0 0}, upload=486909]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:52:33.568743710 (555h23m33.439826745s) [upload=486909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:52:33.568749612 (555h23m33.439832647s) [upload=486909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:52:33.854028773 (555h23m33.725111808s) [upload=486909]: Done uploading segments... Dec 28 18:52:33.854047769 (555h23m33.725130814s) [upload=486909]: Upload finished. err: <nil> Duration: 30.00316779s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.X........................X.....................X.......] duration: 35.087035738s trace: https://jaeger.dev.storj.io/trace/659b4940275e0f05 FAIL (2025-12-28 18:52:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== begin upload segment. Dec 28 18:45:25.180886421 (555h16m25.051969456s) [noise=true , address=145.79.227.21:4490 , piece_id=CLVXZXJRINCX6ZQP2CNAHLXRAOJI2WZUNAP4LSE2QZ5Y4OJ4AELA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.181484279 (555h16m25.052567314s) [noise=true , address=145.79.227.30:4768 , piece_id=GPXQOBUIPLRR3MAPDFDZUTL3D35BDNZIG573G4DAUGUH7DJU5T5A, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.181903237 (555h16m25.052986282s) [noise=true , address=77.90.31.15:4711 , piece_id=UPAPHGHWJBSBJYIDCN2FI3Z46T2JFUVAQOJPQPGTY4OHIWGG5VPA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182209721 (555h16m25.053292766s) [noise=true , address=64.49.13.27:4963 , piece_id=GRQVBULDQLYHV5ILRPMIWTMWP7DPPSJSKICIWEG4EKUXB7PZ7LSA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182531413 (555h16m25.053614448s) [noise=true , address=145.79.227.43:4160 , piece_id=X42ZZOWQLDUD34IWTT7R66VDHAJGA27IEUTAIMURDF4ELHLIW5PQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182972543 (555h16m25.054055578s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183286231 (555h16m25.054369276s) [noise=true , address=45.91.116.8:4051 , piece_id=AY4G65PDYI34KUS4IAUYIK5WCU3VUMN7XCQJVZTXFXWDMGGYNPBQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183597022 (555h16m25.054680057s) [noise=true , address=77.90.31.21:4895 , piece_id=L5U5SMLBF4R3V7BUZLXRKQ2AEGIQGHPNEJL3FGGLF7DF5V2T4UIA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183869430 (555h16m25.054952475s) [noise=true , address=145.79.227.40:4057 , piece_id=QRTNQJUDFQIOCWSZ7CD5OB5ZE4UJ7W2LKKITFJCS7GJ6G2EIAVWQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.185961945 (555h16m25.05704499s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.186737572 (555h16m25.057820617s) [noise=true , address=77.90.31.13:4630 , piece_id=63OQQKRAHKDKQ7PVN4UAIESK72NW6WOH3Q3QMAVZKSS32PATVKFQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.186816813 (555h16m25.057899848s) [noise=true , address=193.149.164.143:4696 , piece_id=4RTCBEIBUFN72SFD4VQ6UDCBEAU4DRF56HJIZT4L76ETKJUBWTZA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187247843 (555h16m25.058330878s) [noise=true , address=45.91.116.9:4281 , piece_id=AIALW5EEA2VRVELDNWXI64XNNUFQOGZ6424CBLXIOK4AD5S25ZSA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187406526 (555h16m25.058489561s) [noise=true , address=45.91.116.11:4130 , piece_id=ALKG6F653PTF6QYD3NTENX5VR2SE3HYTK4KQN6JF5AI2W5GII3CA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187824422 (555h16m25.058907467s) [noise=true , address=145.79.227.44:4208 , piece_id=2SPAW6GFEDZWNDUCMWO6DNDYFVDGBJOL6XGQEFNTZBZNO7XPKXGQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188212400 (555h16m25.059295445s) [noise=true , address=45.91.116.4:4949 , piece_id=E745VQGK2CVZ2ETETNKIQWDODF4K6U7MDZKGW6SBHQZJEDIV2UXQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188670883 (555h16m25.059753928s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188933434 (555h16m25.060016479s) [noise=true , address=77.90.31.18:4835 , piece_id=OQSXAPKIWHOMPWTP3A2LQ5B7DEI53CGPFLTLPZ4BDDOLZ3VVB5LA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:26.071888944 (555h16m25.942971989s) [noise=true , address=193.149.164.143:4696 , piece_id=4RTCBEIBUFN72SFD4VQ6UDCBEAU4DRF56HJIZT4L76ETKJUBWTZA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: storage node overloaded, request limit: 2000 Dec 28 18:45:26.652316958 (555h16m26.523400013s) [upload=486812]: Next returned nil segment Dec 28 18:45:26.652335714 (555h16m26.523418759s) [upload=486812]: Waiting for error group managing segments... Dec 28 18:45:27.070875407 (555h16m26.941958442s) [noise=true , address=145.79.227.21:4490 , piece_id=CLVXZXJRINCX6ZQP2CNAHLXRAOJI2WZUNAP4LSE2QZ5Y4OJ4AELA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.217018568 (555h16m27.088101603s) [noise=true , address=77.90.31.15:4711 , piece_id=UPAPHGHWJBSBJYIDCN2FI3Z46T2JFUVAQOJPQPGTY4OHIWGG5VPA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.244232238 (555h16m27.115315273s) [noise=true , address=77.90.31.21:4895 , piece_id=L5U5SMLBF4R3V7BUZLXRKQ2AEGIQGHPNEJL3FGGLF7DF5V2T4UIA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.264578651 (555h16m27.135661696s) [noise=true , address=77.90.31.13:4630 , piece_id=63OQQKRAHKDKQ7PVN4UAIESK72NW6WOH3Q3QMAVZKSS32PATVKFQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.277932088 (555h16m27.149015134s) [noise=true , address=45.91.116.9:4281 , piece_id=AIALW5EEA2VRVELDNWXI64XNNUFQOGZ6424CBLXIOK4AD5S25ZSA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.324815542 (555h16m27.195898577s) [noise=true , address=45.91.116.8:4051 , piece_id=AY4G65PDYI34KUS4IAUYIK5WCU3VUMN7XCQJVZTXFXWDMGGYNPBQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.329504170 (555h16m27.200587205s) [noise=true , address=77.90.31.18:4835 , piece_id=OQSXAPKIWHOMPWTP3A2LQ5B7DEI53CGPFLTLPZ4BDDOLZ3VVB5LA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.334083971 (555h16m27.205167006s) [noise=true , address=45.91.116.4:4949 , piece_id=E745VQGK2CVZ2ETETNKIQWDODF4K6U7MDZKGW6SBHQZJEDIV2UXQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.489154753 (555h16m27.360237798s) [noise=true , address=64.49.13.27:4963 , piece_id=GRQVBULDQLYHV5ILRPMIWTMWP7DPPSJSKICIWEG4EKUXB7PZ7LSA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.919436724 (555h16m27.790519759s) [noise=true , address=145.79.227.43:4160 , piece_id=X42ZZOWQLDUD34IWTT7R66VDHAJGA27IEUTAIMURDF4ELHLIW5PQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.987848773 (555h16m27.858931818s) [noise=true , address=145.79.227.30:4768 , piece_id=GPXQOBUIPLRR3MAPDFDZUTL3D35BDNZIG573G4DAUGUH7DJU5T5A, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.988333386 (555h16m27.859416421s) [noise=true , address=145.79.227.44:4208 , piece_id=2SPAW6GFEDZWNDUCMWO6DNDYFVDGBJOL6XGQEFNTZBZNO7XPKXGQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.044342256 (555h16m27.915425291s) [noise=true , address=145.79.227.40:4057 , piece_id=QRTNQJUDFQIOCWSZ7CD5OB5ZE4UJ7W2LKKITFJCS7GJ6G2EIAVWQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.051425193 (555h16m27.922508238s) [noise=true , address=45.91.116.11:4130 , piece_id=ALKG6F653PTF6QYD3NTENX5VR2SE3HYTK4KQN6JF5AI2W5GII3CA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.051449089 (555h16m27.922532144s) [seg_pos={0 0}, upload=486812]: Segment reached optimal threshold of 14 pieces. Dec 28 18:45:28.051675730 (555h16m27.922758775s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051686711 (555h16m27.922769756s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051694917 (555h16m27.922777952s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051704274 (555h16m27.922787309s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051813111 (555h16m27.922896146s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1brZ4RtcK5J3xCPmnbsTccF9hrnj8tqhV1WyMqqrjuitgSGMou): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051818422 (555h16m27.922901457s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051850843 (555h16m27.922933878s) [seg_pos={0 0}, upload=486812]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:45:28.051941646 (555h16m27.923024682s) [upload=486812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:45:28.051946335 (555h16m27.92302937s) [upload=486812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:45:28.356338310 (555h16m28.227421345s) [upload=486812]: Done uploading segments... Dec 28 18:45:28.356361995 (555h16m28.22744504s) [upload=486812]: Upload finished. err: <nil> Duration: 35.087071067s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X..X..X....X.X.X.XX..X..X..X...X...XX.......X.X...X.....] duration: 6.636932168s trace: https://jaeger.dev.storj.io/trace/1d60d5d8087e6db7 FAIL (2025-12-28 18:51:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A) context deadline exceeded ======LOGS====== 25.850300317 (555h18m25.721383352s) [upload=486840]: Starting upload Dec 28 18:47:25.850388406 (555h18m25.721471441s) [upload=486840]: Done starting upload Dec 28 18:47:25.850510057 (555h18m25.721593092s) [upload=486840]: Uploading segments... Dec 28 18:47:25.850605819 (555h18m25.721688854s) [upload=486840]: Got next segment. Inline: false Dec 28 18:47:25.850634504 (555h18m25.721717539s) [upload=486840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:47:25.850638311 (555h18m25.721721346s) [upload=486840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:47:26.095877088 (555h18m25.966960123s) [seg_pos={0 0}, upload=486840]: Begin upload segment... Dec 28 18:47:26.095932344 (555h18m25.967015389s) [seg_pos={0 0}, upload=486840]: Done begin upload segment. Dec 28 18:47:26.095978802 (555h18m25.967061837s) [noise=true , address=145.79.227.50:4374 , piece_id=AGE3V6L3TNFLVUPLWXVTLKX7A2GDMTKKF6BR5GM4Y2PVLIE7SJPA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096017486 (555h18m25.967100541s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096296628 (555h18m25.967379673s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096691790 (555h18m25.967774835s) [noise=true , address=145.79.227.19:4078 , piece_id=6GMFV2W7SQS4Y44FERO5MZXF6TKEJZHCY3YMSZD367UMXGJR73IQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097015496 (555h18m25.968098531s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097367847 (555h18m25.968450882s) [noise=true , address=145.79.227.45:4231 , piece_id=E2FBWYQWNFJWN6ZVBDT5KJMLJ3JGM735EWVH6FGZRGFOGQ3LXGHA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097651647 (555h18m25.968734682s) [noise=true , address=77.90.31.18:4805 , piece_id=C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098021793 (555h18m25.969104838s) [noise=true , address=145.79.227.12:4767 , piece_id=CZOKOTYGOMGQTFS3RSQRJTHX3DC6IT6LVVFFRVVZ72BK4BUIB3OA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098453214 (555h18m25.969536249s) [noise=true , address=145.79.227.37:4565 , piece_id=BHDEXF2TMQ6TLMEPP6W6CSER5LZTIXUMHSFT3GKKZ3J42XUNLHGQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098930623 (555h18m25.970013668s) [noise=true , address=45.91.116.13:4156 , piece_id=Q36RL5NSW7PSZMPR2GWW7JSL6RCR5VRJBAXT27BEE6W256XDDNNQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.099392252 (555h18m25.970475287s) [noise=true , address=45.91.116.6:4974 , piece_id=63MZ5OJ6BIMZXHFLMCTCNZSSCIZEX7UEXS33WRPJHX4RFBQZLJWQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.100755298 (555h18m25.971838333s) [noise=true , address=77.90.31.16:4736 , piece_id=NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.100928328 (555h18m25.972011373s) [noise=true , address=145.79.227.2:4113 , piece_id=BP4TRNX775AR7IAYCYIEJC6NRNVUWO6J232STFROA6D42U7JI62Q, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.103054576 (555h18m25.974137621s) [noise=true , address=145.79.227.51:4411 , piece_id=NXII74KWEFEFHYHUS6AY2QTKQ3T32AUXLLQI7HUHQY3R7WRLKVXQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.104831500 (555h18m25.975914545s) [noise=true , address=45.91.116.9:4263 , piece_id=7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.106317308 (555h18m25.977400353s) [noise=true , address=145.79.227.11:4730 , piece_id=N4Y5TMAMK4JK33XPETURRJCLTHVKZSHMAYMOQJPM57QWBJAJCGOQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:27.818567118 (555h18m27.689650153s) [upload=486840]: Next returned nil segment Dec 28 18:47:27.818578900 (555h18m27.689661945s) [upload=486840]: Waiting for error group managing segments... Dec 28 18:47:28.101764976 (555h18m27.972848022s) [noise=true , address=145.79.227.50:4374 , piece_id=AGE3V6L3TNFLVUPLWXVTLKX7A2GDMTKKF6BR5GM4Y2PVLIE7SJPA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.102330152 (555h18m27.973413198s) [noise=true , address=145.79.227.45:4231 , piece_id=E2FBWYQWNFJWN6ZVBDT5KJMLJ3JGM735EWVH6FGZRGFOGQ3LXGHA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.104756042 (555h18m27.975839117s) [noise=true , address=145.79.227.51:4411 , piece_id=NXII74KWEFEFHYHUS6AY2QTKQ3T32AUXLLQI7HUHQY3R7WRLKVXQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.105995141 (555h18m27.977078186s) [noise=true , address=145.79.227.19:4078 , piece_id=6GMFV2W7SQS4Y44FERO5MZXF6TKEJZHCY3YMSZD367UMXGJR73IQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.108319868 (555h18m27.979402913s) [noise=true , address=145.79.227.12:4767 , piece_id=CZOKOTYGOMGQTFS3RSQRJTHX3DC6IT6LVVFFRVVZ72BK4BUIB3OA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.119741496 (555h18m27.990824542s) [noise=true , address=145.79.227.37:4565 , piece_id=BHDEXF2TMQ6TLMEPP6W6CSER5LZTIXUMHSFT3GKKZ3J42XUNLHGQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.125589121 (555h18m27.996672156s) [noise=true , address=145.79.227.2:4113 , piece_id=BP4TRNX775AR7IAYCYIEJC6NRNVUWO6J232STFROA6D42U7JI62Q, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.135538287 (555h18m28.006621322s) [noise=true , address=145.79.227.11:4730 , piece_id=N4Y5TMAMK4JK33XPETURRJCLTHVKZSHMAYMOQJPM57QWBJAJCGOQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.317845352 (555h18m28.188928397s) [noise=true , address=77.90.31.16:4736 , piece_id=NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.372520352 (555h18m28.243603387s) [noise=true , address=45.91.116.13:4156 , piece_id=Q36RL5NSW7PSZMPR2GWW7JSL6RCR5VRJBAXT27BEE6W256XDDNNQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.377415213 (555h18m28.248498258s) [noise=true , address=45.91.116.9:4263 , piece_id=7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.384564486 (555h18m28.255647521s) [noise=true , address=77.90.31.18:4805 , piece_id=C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.451238624 (555h18m28.322321659s) [noise=true , address=45.91.116.6:4974 , piece_id=63MZ5OJ6BIMZXHFLMCTCNZSSCIZEX7UEXS33WRPJHX4RFBQZLJWQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.451258964 (555h18m28.322341999s) [seg_pos={0 0}, upload=486840]: Segment reached optimal threshold of 13 pieces. Dec 28 18:47:28.451443845 (555h18m28.32252688s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection Dec 28 18:47:28.451450207 (555h18m28.322533242s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451566598 (555h18m28.322649634s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 18:47:28.451582018 (555h18m28.322665053s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451602367 (555h18m28.322685412s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 28 18:47:28.451608830 (555h18m28.322691865s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451624790 (555h18m28.322707825s) [seg_pos={0 0}, upload=486840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:47:28.451690184 (555h18m28.322773219s) [upload=486840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:47:28.451695344 (555h18m28.322778379s) [upload=486840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:47:28.747747940 (555h18m28.618830975s) [upload=486840]: Done uploading segments... Dec 28 18:47:28.747767095 (555h18m28.618850131s) [upload=486840]: Upload finished. err: <nil> Duration: 6.636954449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X....X..X..X....X.X.X.XX..X..X..X...X...XX.......X.X...X....] duration: 16.248029641s trace: https://jaeger.dev.storj.io/trace/1d60d5d8087e6db7 FAIL (2025-12-28 18:51:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A) context deadline exceeded ======LOGS====== 25.850300317 (555h18m25.721383352s) [upload=486840]: Starting upload Dec 28 18:47:25.850388406 (555h18m25.721471441s) [upload=486840]: Done starting upload Dec 28 18:47:25.850510057 (555h18m25.721593092s) [upload=486840]: Uploading segments... Dec 28 18:47:25.850605819 (555h18m25.721688854s) [upload=486840]: Got next segment. Inline: false Dec 28 18:47:25.850634504 (555h18m25.721717539s) [upload=486840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:47:25.850638311 (555h18m25.721721346s) [upload=486840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:47:26.095877088 (555h18m25.966960123s) [seg_pos={0 0}, upload=486840]: Begin upload segment... Dec 28 18:47:26.095932344 (555h18m25.967015389s) [seg_pos={0 0}, upload=486840]: Done begin upload segment. Dec 28 18:47:26.095978802 (555h18m25.967061837s) [noise=true , address=145.79.227.50:4374 , piece_id=AGE3V6L3TNFLVUPLWXVTLKX7A2GDMTKKF6BR5GM4Y2PVLIE7SJPA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096017486 (555h18m25.967100541s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096296628 (555h18m25.967379673s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096691790 (555h18m25.967774835s) [noise=true , address=145.79.227.19:4078 , piece_id=6GMFV2W7SQS4Y44FERO5MZXF6TKEJZHCY3YMSZD367UMXGJR73IQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097015496 (555h18m25.968098531s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097367847 (555h18m25.968450882s) [noise=true , address=145.79.227.45:4231 , piece_id=E2FBWYQWNFJWN6ZVBDT5KJMLJ3JGM735EWVH6FGZRGFOGQ3LXGHA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097651647 (555h18m25.968734682s) [noise=true , address=77.90.31.18:4805 , piece_id=C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098021793 (555h18m25.969104838s) [noise=true , address=145.79.227.12:4767 , piece_id=CZOKOTYGOMGQTFS3RSQRJTHX3DC6IT6LVVFFRVVZ72BK4BUIB3OA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098453214 (555h18m25.969536249s) [noise=true , address=145.79.227.37:4565 , piece_id=BHDEXF2TMQ6TLMEPP6W6CSER5LZTIXUMHSFT3GKKZ3J42XUNLHGQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098930623 (555h18m25.970013668s) [noise=true , address=45.91.116.13:4156 , piece_id=Q36RL5NSW7PSZMPR2GWW7JSL6RCR5VRJBAXT27BEE6W256XDDNNQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.099392252 (555h18m25.970475287s) [noise=true , address=45.91.116.6:4974 , piece_id=63MZ5OJ6BIMZXHFLMCTCNZSSCIZEX7UEXS33WRPJHX4RFBQZLJWQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.100755298 (555h18m25.971838333s) [noise=true , address=77.90.31.16:4736 , piece_id=NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.100928328 (555h18m25.972011373s) [noise=true , address=145.79.227.2:4113 , piece_id=BP4TRNX775AR7IAYCYIEJC6NRNVUWO6J232STFROA6D42U7JI62Q, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.103054576 (555h18m25.974137621s) [noise=true , address=145.79.227.51:4411 , piece_id=NXII74KWEFEFHYHUS6AY2QTKQ3T32AUXLLQI7HUHQY3R7WRLKVXQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.104831500 (555h18m25.975914545s) [noise=true , address=45.91.116.9:4263 , piece_id=7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.106317308 (555h18m25.977400353s) [noise=true , address=145.79.227.11:4730 , piece_id=N4Y5TMAMK4JK33XPETURRJCLTHVKZSHMAYMOQJPM57QWBJAJCGOQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:27.818567118 (555h18m27.689650153s) [upload=486840]: Next returned nil segment Dec 28 18:47:27.818578900 (555h18m27.689661945s) [upload=486840]: Waiting for error group managing segments... Dec 28 18:47:28.101764976 (555h18m27.972848022s) [noise=true , address=145.79.227.50:4374 , piece_id=AGE3V6L3TNFLVUPLWXVTLKX7A2GDMTKKF6BR5GM4Y2PVLIE7SJPA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.102330152 (555h18m27.973413198s) [noise=true , address=145.79.227.45:4231 , piece_id=E2FBWYQWNFJWN6ZVBDT5KJMLJ3JGM735EWVH6FGZRGFOGQ3LXGHA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.104756042 (555h18m27.975839117s) [noise=true , address=145.79.227.51:4411 , piece_id=NXII74KWEFEFHYHUS6AY2QTKQ3T32AUXLLQI7HUHQY3R7WRLKVXQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.105995141 (555h18m27.977078186s) [noise=true , address=145.79.227.19:4078 , piece_id=6GMFV2W7SQS4Y44FERO5MZXF6TKEJZHCY3YMSZD367UMXGJR73IQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.108319868 (555h18m27.979402913s) [noise=true , address=145.79.227.12:4767 , piece_id=CZOKOTYGOMGQTFS3RSQRJTHX3DC6IT6LVVFFRVVZ72BK4BUIB3OA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.119741496 (555h18m27.990824542s) [noise=true , address=145.79.227.37:4565 , piece_id=BHDEXF2TMQ6TLMEPP6W6CSER5LZTIXUMHSFT3GKKZ3J42XUNLHGQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.125589121 (555h18m27.996672156s) [noise=true , address=145.79.227.2:4113 , piece_id=BP4TRNX775AR7IAYCYIEJC6NRNVUWO6J232STFROA6D42U7JI62Q, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.135538287 (555h18m28.006621322s) [noise=true , address=145.79.227.11:4730 , piece_id=N4Y5TMAMK4JK33XPETURRJCLTHVKZSHMAYMOQJPM57QWBJAJCGOQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.317845352 (555h18m28.188928397s) [noise=true , address=77.90.31.16:4736 , piece_id=NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.372520352 (555h18m28.243603387s) [noise=true , address=45.91.116.13:4156 , piece_id=Q36RL5NSW7PSZMPR2GWW7JSL6RCR5VRJBAXT27BEE6W256XDDNNQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.377415213 (555h18m28.248498258s) [noise=true , address=45.91.116.9:4263 , piece_id=7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.384564486 (555h18m28.255647521s) [noise=true , address=77.90.31.18:4805 , piece_id=C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.451238624 (555h18m28.322321659s) [noise=true , address=45.91.116.6:4974 , piece_id=63MZ5OJ6BIMZXHFLMCTCNZSSCIZEX7UEXS33WRPJHX4RFBQZLJWQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.451258964 (555h18m28.322341999s) [seg_pos={0 0}, upload=486840]: Segment reached optimal threshold of 13 pieces. Dec 28 18:47:28.451443845 (555h18m28.32252688s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection Dec 28 18:47:28.451450207 (555h18m28.322533242s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451566598 (555h18m28.322649634s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 18:47:28.451582018 (555h18m28.322665053s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451602367 (555h18m28.322685412s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 28 18:47:28.451608830 (555h18m28.322691865s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451624790 (555h18m28.322707825s) [seg_pos={0 0}, upload=486840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:47:28.451690184 (555h18m28.322773219s) [upload=486840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:47:28.451695344 (555h18m28.322778379s) [upload=486840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:47:28.747747940 (555h18m28.618830975s) [upload=486840]: Done uploading segments... Dec 28 18:47:28.747767095 (555h18m28.618850131s) [upload=486840]: Upload finished. err: <nil> Duration: 16.248054277s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.X........................X.....................X......] duration: 25.254840336s trace: https://jaeger.dev.storj.io/trace/659b4940275e0f05 FAIL (2025-12-28 18:51:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== begin upload segment. Dec 28 18:45:25.180886421 (555h16m25.051969456s) [noise=true , address=145.79.227.21:4490 , piece_id=CLVXZXJRINCX6ZQP2CNAHLXRAOJI2WZUNAP4LSE2QZ5Y4OJ4AELA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.181484279 (555h16m25.052567314s) [noise=true , address=145.79.227.30:4768 , piece_id=GPXQOBUIPLRR3MAPDFDZUTL3D35BDNZIG573G4DAUGUH7DJU5T5A, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.181903237 (555h16m25.052986282s) [noise=true , address=77.90.31.15:4711 , piece_id=UPAPHGHWJBSBJYIDCN2FI3Z46T2JFUVAQOJPQPGTY4OHIWGG5VPA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182209721 (555h16m25.053292766s) [noise=true , address=64.49.13.27:4963 , piece_id=GRQVBULDQLYHV5ILRPMIWTMWP7DPPSJSKICIWEG4EKUXB7PZ7LSA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182531413 (555h16m25.053614448s) [noise=true , address=145.79.227.43:4160 , piece_id=X42ZZOWQLDUD34IWTT7R66VDHAJGA27IEUTAIMURDF4ELHLIW5PQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182972543 (555h16m25.054055578s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183286231 (555h16m25.054369276s) [noise=true , address=45.91.116.8:4051 , piece_id=AY4G65PDYI34KUS4IAUYIK5WCU3VUMN7XCQJVZTXFXWDMGGYNPBQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183597022 (555h16m25.054680057s) [noise=true , address=77.90.31.21:4895 , piece_id=L5U5SMLBF4R3V7BUZLXRKQ2AEGIQGHPNEJL3FGGLF7DF5V2T4UIA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183869430 (555h16m25.054952475s) [noise=true , address=145.79.227.40:4057 , piece_id=QRTNQJUDFQIOCWSZ7CD5OB5ZE4UJ7W2LKKITFJCS7GJ6G2EIAVWQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.185961945 (555h16m25.05704499s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.186737572 (555h16m25.057820617s) [noise=true , address=77.90.31.13:4630 , piece_id=63OQQKRAHKDKQ7PVN4UAIESK72NW6WOH3Q3QMAVZKSS32PATVKFQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.186816813 (555h16m25.057899848s) [noise=true , address=193.149.164.143:4696 , piece_id=4RTCBEIBUFN72SFD4VQ6UDCBEAU4DRF56HJIZT4L76ETKJUBWTZA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187247843 (555h16m25.058330878s) [noise=true , address=45.91.116.9:4281 , piece_id=AIALW5EEA2VRVELDNWXI64XNNUFQOGZ6424CBLXIOK4AD5S25ZSA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187406526 (555h16m25.058489561s) [noise=true , address=45.91.116.11:4130 , piece_id=ALKG6F653PTF6QYD3NTENX5VR2SE3HYTK4KQN6JF5AI2W5GII3CA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187824422 (555h16m25.058907467s) [noise=true , address=145.79.227.44:4208 , piece_id=2SPAW6GFEDZWNDUCMWO6DNDYFVDGBJOL6XGQEFNTZBZNO7XPKXGQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188212400 (555h16m25.059295445s) [noise=true , address=45.91.116.4:4949 , piece_id=E745VQGK2CVZ2ETETNKIQWDODF4K6U7MDZKGW6SBHQZJEDIV2UXQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188670883 (555h16m25.059753928s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188933434 (555h16m25.060016479s) [noise=true , address=77.90.31.18:4835 , piece_id=OQSXAPKIWHOMPWTP3A2LQ5B7DEI53CGPFLTLPZ4BDDOLZ3VVB5LA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:26.071888944 (555h16m25.942971989s) [noise=true , address=193.149.164.143:4696 , piece_id=4RTCBEIBUFN72SFD4VQ6UDCBEAU4DRF56HJIZT4L76ETKJUBWTZA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: storage node overloaded, request limit: 2000 Dec 28 18:45:26.652316958 (555h16m26.523400013s) [upload=486812]: Next returned nil segment Dec 28 18:45:26.652335714 (555h16m26.523418759s) [upload=486812]: Waiting for error group managing segments... Dec 28 18:45:27.070875407 (555h16m26.941958442s) [noise=true , address=145.79.227.21:4490 , piece_id=CLVXZXJRINCX6ZQP2CNAHLXRAOJI2WZUNAP4LSE2QZ5Y4OJ4AELA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.217018568 (555h16m27.088101603s) [noise=true , address=77.90.31.15:4711 , piece_id=UPAPHGHWJBSBJYIDCN2FI3Z46T2JFUVAQOJPQPGTY4OHIWGG5VPA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.244232238 (555h16m27.115315273s) [noise=true , address=77.90.31.21:4895 , piece_id=L5U5SMLBF4R3V7BUZLXRKQ2AEGIQGHPNEJL3FGGLF7DF5V2T4UIA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.264578651 (555h16m27.135661696s) [noise=true , address=77.90.31.13:4630 , piece_id=63OQQKRAHKDKQ7PVN4UAIESK72NW6WOH3Q3QMAVZKSS32PATVKFQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.277932088 (555h16m27.149015134s) [noise=true , address=45.91.116.9:4281 , piece_id=AIALW5EEA2VRVELDNWXI64XNNUFQOGZ6424CBLXIOK4AD5S25ZSA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.324815542 (555h16m27.195898577s) [noise=true , address=45.91.116.8:4051 , piece_id=AY4G65PDYI34KUS4IAUYIK5WCU3VUMN7XCQJVZTXFXWDMGGYNPBQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.329504170 (555h16m27.200587205s) [noise=true , address=77.90.31.18:4835 , piece_id=OQSXAPKIWHOMPWTP3A2LQ5B7DEI53CGPFLTLPZ4BDDOLZ3VVB5LA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.334083971 (555h16m27.205167006s) [noise=true , address=45.91.116.4:4949 , piece_id=E745VQGK2CVZ2ETETNKIQWDODF4K6U7MDZKGW6SBHQZJEDIV2UXQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.489154753 (555h16m27.360237798s) [noise=true , address=64.49.13.27:4963 , piece_id=GRQVBULDQLYHV5ILRPMIWTMWP7DPPSJSKICIWEG4EKUXB7PZ7LSA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.919436724 (555h16m27.790519759s) [noise=true , address=145.79.227.43:4160 , piece_id=X42ZZOWQLDUD34IWTT7R66VDHAJGA27IEUTAIMURDF4ELHLIW5PQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.987848773 (555h16m27.858931818s) [noise=true , address=145.79.227.30:4768 , piece_id=GPXQOBUIPLRR3MAPDFDZUTL3D35BDNZIG573G4DAUGUH7DJU5T5A, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.988333386 (555h16m27.859416421s) [noise=true , address=145.79.227.44:4208 , piece_id=2SPAW6GFEDZWNDUCMWO6DNDYFVDGBJOL6XGQEFNTZBZNO7XPKXGQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.044342256 (555h16m27.915425291s) [noise=true , address=145.79.227.40:4057 , piece_id=QRTNQJUDFQIOCWSZ7CD5OB5ZE4UJ7W2LKKITFJCS7GJ6G2EIAVWQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.051425193 (555h16m27.922508238s) [noise=true , address=45.91.116.11:4130 , piece_id=ALKG6F653PTF6QYD3NTENX5VR2SE3HYTK4KQN6JF5AI2W5GII3CA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.051449089 (555h16m27.922532144s) [seg_pos={0 0}, upload=486812]: Segment reached optimal threshold of 14 pieces. Dec 28 18:45:28.051675730 (555h16m27.922758775s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051686711 (555h16m27.922769756s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051694917 (555h16m27.922777952s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051704274 (555h16m27.922787309s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051813111 (555h16m27.922896146s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1brZ4RtcK5J3xCPmnbsTccF9hrnj8tqhV1WyMqqrjuitgSGMou): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051818422 (555h16m27.922901457s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051850843 (555h16m27.922933878s) [seg_pos={0 0}, upload=486812]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:45:28.051941646 (555h16m27.923024682s) [upload=486812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:45:28.051946335 (555h16m27.92302937s) [upload=486812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:45:28.356338310 (555h16m28.227421345s) [upload=486812]: Done uploading segments... Dec 28 18:45:28.356361995 (555h16m28.22744504s) [upload=486812]: Upload finished. err: <nil> Duration: 25.254864922s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X....X..X..X....X.X.X.XX..X..X..X...X...XX.......X.X...X...] duration: 6.021681141s trace: https://jaeger.dev.storj.io/trace/1d60d5d8087e6db7 FAIL (2025-12-28 18:50:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A) context deadline exceeded ======LOGS====== 25.850300317 (555h18m25.721383352s) [upload=486840]: Starting upload Dec 28 18:47:25.850388406 (555h18m25.721471441s) [upload=486840]: Done starting upload Dec 28 18:47:25.850510057 (555h18m25.721593092s) [upload=486840]: Uploading segments... Dec 28 18:47:25.850605819 (555h18m25.721688854s) [upload=486840]: Got next segment. Inline: false Dec 28 18:47:25.850634504 (555h18m25.721717539s) [upload=486840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:47:25.850638311 (555h18m25.721721346s) [upload=486840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:47:26.095877088 (555h18m25.966960123s) [seg_pos={0 0}, upload=486840]: Begin upload segment... Dec 28 18:47:26.095932344 (555h18m25.967015389s) [seg_pos={0 0}, upload=486840]: Done begin upload segment. Dec 28 18:47:26.095978802 (555h18m25.967061837s) [noise=true , address=145.79.227.50:4374 , piece_id=AGE3V6L3TNFLVUPLWXVTLKX7A2GDMTKKF6BR5GM4Y2PVLIE7SJPA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096017486 (555h18m25.967100541s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096296628 (555h18m25.967379673s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096691790 (555h18m25.967774835s) [noise=true , address=145.79.227.19:4078 , piece_id=6GMFV2W7SQS4Y44FERO5MZXF6TKEJZHCY3YMSZD367UMXGJR73IQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097015496 (555h18m25.968098531s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097367847 (555h18m25.968450882s) [noise=true , address=145.79.227.45:4231 , piece_id=E2FBWYQWNFJWN6ZVBDT5KJMLJ3JGM735EWVH6FGZRGFOGQ3LXGHA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097651647 (555h18m25.968734682s) [noise=true , address=77.90.31.18:4805 , piece_id=C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098021793 (555h18m25.969104838s) [noise=true , address=145.79.227.12:4767 , piece_id=CZOKOTYGOMGQTFS3RSQRJTHX3DC6IT6LVVFFRVVZ72BK4BUIB3OA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098453214 (555h18m25.969536249s) [noise=true , address=145.79.227.37:4565 , piece_id=BHDEXF2TMQ6TLMEPP6W6CSER5LZTIXUMHSFT3GKKZ3J42XUNLHGQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098930623 (555h18m25.970013668s) [noise=true , address=45.91.116.13:4156 , piece_id=Q36RL5NSW7PSZMPR2GWW7JSL6RCR5VRJBAXT27BEE6W256XDDNNQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.099392252 (555h18m25.970475287s) [noise=true , address=45.91.116.6:4974 , piece_id=63MZ5OJ6BIMZXHFLMCTCNZSSCIZEX7UEXS33WRPJHX4RFBQZLJWQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.100755298 (555h18m25.971838333s) [noise=true , address=77.90.31.16:4736 , piece_id=NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.100928328 (555h18m25.972011373s) [noise=true , address=145.79.227.2:4113 , piece_id=BP4TRNX775AR7IAYCYIEJC6NRNVUWO6J232STFROA6D42U7JI62Q, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.103054576 (555h18m25.974137621s) [noise=true , address=145.79.227.51:4411 , piece_id=NXII74KWEFEFHYHUS6AY2QTKQ3T32AUXLLQI7HUHQY3R7WRLKVXQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.104831500 (555h18m25.975914545s) [noise=true , address=45.91.116.9:4263 , piece_id=7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.106317308 (555h18m25.977400353s) [noise=true , address=145.79.227.11:4730 , piece_id=N4Y5TMAMK4JK33XPETURRJCLTHVKZSHMAYMOQJPM57QWBJAJCGOQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:27.818567118 (555h18m27.689650153s) [upload=486840]: Next returned nil segment Dec 28 18:47:27.818578900 (555h18m27.689661945s) [upload=486840]: Waiting for error group managing segments... Dec 28 18:47:28.101764976 (555h18m27.972848022s) [noise=true , address=145.79.227.50:4374 , piece_id=AGE3V6L3TNFLVUPLWXVTLKX7A2GDMTKKF6BR5GM4Y2PVLIE7SJPA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.102330152 (555h18m27.973413198s) [noise=true , address=145.79.227.45:4231 , piece_id=E2FBWYQWNFJWN6ZVBDT5KJMLJ3JGM735EWVH6FGZRGFOGQ3LXGHA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.104756042 (555h18m27.975839117s) [noise=true , address=145.79.227.51:4411 , piece_id=NXII74KWEFEFHYHUS6AY2QTKQ3T32AUXLLQI7HUHQY3R7WRLKVXQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.105995141 (555h18m27.977078186s) [noise=true , address=145.79.227.19:4078 , piece_id=6GMFV2W7SQS4Y44FERO5MZXF6TKEJZHCY3YMSZD367UMXGJR73IQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.108319868 (555h18m27.979402913s) [noise=true , address=145.79.227.12:4767 , piece_id=CZOKOTYGOMGQTFS3RSQRJTHX3DC6IT6LVVFFRVVZ72BK4BUIB3OA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.119741496 (555h18m27.990824542s) [noise=true , address=145.79.227.37:4565 , piece_id=BHDEXF2TMQ6TLMEPP6W6CSER5LZTIXUMHSFT3GKKZ3J42XUNLHGQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.125589121 (555h18m27.996672156s) [noise=true , address=145.79.227.2:4113 , piece_id=BP4TRNX775AR7IAYCYIEJC6NRNVUWO6J232STFROA6D42U7JI62Q, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.135538287 (555h18m28.006621322s) [noise=true , address=145.79.227.11:4730 , piece_id=N4Y5TMAMK4JK33XPETURRJCLTHVKZSHMAYMOQJPM57QWBJAJCGOQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.317845352 (555h18m28.188928397s) [noise=true , address=77.90.31.16:4736 , piece_id=NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.372520352 (555h18m28.243603387s) [noise=true , address=45.91.116.13:4156 , piece_id=Q36RL5NSW7PSZMPR2GWW7JSL6RCR5VRJBAXT27BEE6W256XDDNNQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.377415213 (555h18m28.248498258s) [noise=true , address=45.91.116.9:4263 , piece_id=7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.384564486 (555h18m28.255647521s) [noise=true , address=77.90.31.18:4805 , piece_id=C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.451238624 (555h18m28.322321659s) [noise=true , address=45.91.116.6:4974 , piece_id=63MZ5OJ6BIMZXHFLMCTCNZSSCIZEX7UEXS33WRPJHX4RFBQZLJWQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.451258964 (555h18m28.322341999s) [seg_pos={0 0}, upload=486840]: Segment reached optimal threshold of 13 pieces. Dec 28 18:47:28.451443845 (555h18m28.32252688s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection Dec 28 18:47:28.451450207 (555h18m28.322533242s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451566598 (555h18m28.322649634s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 18:47:28.451582018 (555h18m28.322665053s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451602367 (555h18m28.322685412s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 28 18:47:28.451608830 (555h18m28.322691865s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451624790 (555h18m28.322707825s) [seg_pos={0 0}, upload=486840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:47:28.451690184 (555h18m28.322773219s) [upload=486840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:47:28.451695344 (555h18m28.322778379s) [upload=486840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:47:28.747747940 (555h18m28.618830975s) [upload=486840]: Done uploading segments... Dec 28 18:47:28.747767095 (555h18m28.618850131s) [upload=486840]: Upload finished. err: <nil> Duration: 6.021725817s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.X........................X.....................X.....] duration: 37.967546699s trace: https://jaeger.dev.storj.io/trace/659b4940275e0f05 FAIL (2025-12-28 18:50:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== begin upload segment. Dec 28 18:45:25.180886421 (555h16m25.051969456s) [noise=true , address=145.79.227.21:4490 , piece_id=CLVXZXJRINCX6ZQP2CNAHLXRAOJI2WZUNAP4LSE2QZ5Y4OJ4AELA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.181484279 (555h16m25.052567314s) [noise=true , address=145.79.227.30:4768 , piece_id=GPXQOBUIPLRR3MAPDFDZUTL3D35BDNZIG573G4DAUGUH7DJU5T5A, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.181903237 (555h16m25.052986282s) [noise=true , address=77.90.31.15:4711 , piece_id=UPAPHGHWJBSBJYIDCN2FI3Z46T2JFUVAQOJPQPGTY4OHIWGG5VPA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182209721 (555h16m25.053292766s) [noise=true , address=64.49.13.27:4963 , piece_id=GRQVBULDQLYHV5ILRPMIWTMWP7DPPSJSKICIWEG4EKUXB7PZ7LSA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182531413 (555h16m25.053614448s) [noise=true , address=145.79.227.43:4160 , piece_id=X42ZZOWQLDUD34IWTT7R66VDHAJGA27IEUTAIMURDF4ELHLIW5PQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182972543 (555h16m25.054055578s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183286231 (555h16m25.054369276s) [noise=true , address=45.91.116.8:4051 , piece_id=AY4G65PDYI34KUS4IAUYIK5WCU3VUMN7XCQJVZTXFXWDMGGYNPBQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183597022 (555h16m25.054680057s) [noise=true , address=77.90.31.21:4895 , piece_id=L5U5SMLBF4R3V7BUZLXRKQ2AEGIQGHPNEJL3FGGLF7DF5V2T4UIA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183869430 (555h16m25.054952475s) [noise=true , address=145.79.227.40:4057 , piece_id=QRTNQJUDFQIOCWSZ7CD5OB5ZE4UJ7W2LKKITFJCS7GJ6G2EIAVWQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.185961945 (555h16m25.05704499s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.186737572 (555h16m25.057820617s) [noise=true , address=77.90.31.13:4630 , piece_id=63OQQKRAHKDKQ7PVN4UAIESK72NW6WOH3Q3QMAVZKSS32PATVKFQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.186816813 (555h16m25.057899848s) [noise=true , address=193.149.164.143:4696 , piece_id=4RTCBEIBUFN72SFD4VQ6UDCBEAU4DRF56HJIZT4L76ETKJUBWTZA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187247843 (555h16m25.058330878s) [noise=true , address=45.91.116.9:4281 , piece_id=AIALW5EEA2VRVELDNWXI64XNNUFQOGZ6424CBLXIOK4AD5S25ZSA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187406526 (555h16m25.058489561s) [noise=true , address=45.91.116.11:4130 , piece_id=ALKG6F653PTF6QYD3NTENX5VR2SE3HYTK4KQN6JF5AI2W5GII3CA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187824422 (555h16m25.058907467s) [noise=true , address=145.79.227.44:4208 , piece_id=2SPAW6GFEDZWNDUCMWO6DNDYFVDGBJOL6XGQEFNTZBZNO7XPKXGQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188212400 (555h16m25.059295445s) [noise=true , address=45.91.116.4:4949 , piece_id=E745VQGK2CVZ2ETETNKIQWDODF4K6U7MDZKGW6SBHQZJEDIV2UXQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188670883 (555h16m25.059753928s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188933434 (555h16m25.060016479s) [noise=true , address=77.90.31.18:4835 , piece_id=OQSXAPKIWHOMPWTP3A2LQ5B7DEI53CGPFLTLPZ4BDDOLZ3VVB5LA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:26.071888944 (555h16m25.942971989s) [noise=true , address=193.149.164.143:4696 , piece_id=4RTCBEIBUFN72SFD4VQ6UDCBEAU4DRF56HJIZT4L76ETKJUBWTZA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: storage node overloaded, request limit: 2000 Dec 28 18:45:26.652316958 (555h16m26.523400013s) [upload=486812]: Next returned nil segment Dec 28 18:45:26.652335714 (555h16m26.523418759s) [upload=486812]: Waiting for error group managing segments... Dec 28 18:45:27.070875407 (555h16m26.941958442s) [noise=true , address=145.79.227.21:4490 , piece_id=CLVXZXJRINCX6ZQP2CNAHLXRAOJI2WZUNAP4LSE2QZ5Y4OJ4AELA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.217018568 (555h16m27.088101603s) [noise=true , address=77.90.31.15:4711 , piece_id=UPAPHGHWJBSBJYIDCN2FI3Z46T2JFUVAQOJPQPGTY4OHIWGG5VPA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.244232238 (555h16m27.115315273s) [noise=true , address=77.90.31.21:4895 , piece_id=L5U5SMLBF4R3V7BUZLXRKQ2AEGIQGHPNEJL3FGGLF7DF5V2T4UIA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.264578651 (555h16m27.135661696s) [noise=true , address=77.90.31.13:4630 , piece_id=63OQQKRAHKDKQ7PVN4UAIESK72NW6WOH3Q3QMAVZKSS32PATVKFQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.277932088 (555h16m27.149015134s) [noise=true , address=45.91.116.9:4281 , piece_id=AIALW5EEA2VRVELDNWXI64XNNUFQOGZ6424CBLXIOK4AD5S25ZSA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.324815542 (555h16m27.195898577s) [noise=true , address=45.91.116.8:4051 , piece_id=AY4G65PDYI34KUS4IAUYIK5WCU3VUMN7XCQJVZTXFXWDMGGYNPBQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.329504170 (555h16m27.200587205s) [noise=true , address=77.90.31.18:4835 , piece_id=OQSXAPKIWHOMPWTP3A2LQ5B7DEI53CGPFLTLPZ4BDDOLZ3VVB5LA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.334083971 (555h16m27.205167006s) [noise=true , address=45.91.116.4:4949 , piece_id=E745VQGK2CVZ2ETETNKIQWDODF4K6U7MDZKGW6SBHQZJEDIV2UXQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.489154753 (555h16m27.360237798s) [noise=true , address=64.49.13.27:4963 , piece_id=GRQVBULDQLYHV5ILRPMIWTMWP7DPPSJSKICIWEG4EKUXB7PZ7LSA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.919436724 (555h16m27.790519759s) [noise=true , address=145.79.227.43:4160 , piece_id=X42ZZOWQLDUD34IWTT7R66VDHAJGA27IEUTAIMURDF4ELHLIW5PQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.987848773 (555h16m27.858931818s) [noise=true , address=145.79.227.30:4768 , piece_id=GPXQOBUIPLRR3MAPDFDZUTL3D35BDNZIG573G4DAUGUH7DJU5T5A, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.988333386 (555h16m27.859416421s) [noise=true , address=145.79.227.44:4208 , piece_id=2SPAW6GFEDZWNDUCMWO6DNDYFVDGBJOL6XGQEFNTZBZNO7XPKXGQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.044342256 (555h16m27.915425291s) [noise=true , address=145.79.227.40:4057 , piece_id=QRTNQJUDFQIOCWSZ7CD5OB5ZE4UJ7W2LKKITFJCS7GJ6G2EIAVWQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.051425193 (555h16m27.922508238s) [noise=true , address=45.91.116.11:4130 , piece_id=ALKG6F653PTF6QYD3NTENX5VR2SE3HYTK4KQN6JF5AI2W5GII3CA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.051449089 (555h16m27.922532144s) [seg_pos={0 0}, upload=486812]: Segment reached optimal threshold of 14 pieces. Dec 28 18:45:28.051675730 (555h16m27.922758775s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051686711 (555h16m27.922769756s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051694917 (555h16m27.922777952s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051704274 (555h16m27.922787309s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051813111 (555h16m27.922896146s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1brZ4RtcK5J3xCPmnbsTccF9hrnj8tqhV1WyMqqrjuitgSGMou): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051818422 (555h16m27.922901457s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051850843 (555h16m27.922933878s) [seg_pos={0 0}, upload=486812]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:45:28.051941646 (555h16m27.923024682s) [upload=486812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:45:28.051946335 (555h16m27.92302937s) [upload=486812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:45:28.356338310 (555h16m28.227421345s) [upload=486812]: Done uploading segments... Dec 28 18:45:28.356361995 (555h16m28.22744504s) [upload=486812]: Upload finished. err: <nil> Duration: 37.967566457s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.X........................X.....................X....] duration: 19.430409402s trace: https://jaeger.dev.storj.io/trace/659b4940275e0f05 FAIL (2025-12-28 18:49:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== begin upload segment. Dec 28 18:45:25.180886421 (555h16m25.051969456s) [noise=true , address=145.79.227.21:4490 , piece_id=CLVXZXJRINCX6ZQP2CNAHLXRAOJI2WZUNAP4LSE2QZ5Y4OJ4AELA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.181484279 (555h16m25.052567314s) [noise=true , address=145.79.227.30:4768 , piece_id=GPXQOBUIPLRR3MAPDFDZUTL3D35BDNZIG573G4DAUGUH7DJU5T5A, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.181903237 (555h16m25.052986282s) [noise=true , address=77.90.31.15:4711 , piece_id=UPAPHGHWJBSBJYIDCN2FI3Z46T2JFUVAQOJPQPGTY4OHIWGG5VPA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182209721 (555h16m25.053292766s) [noise=true , address=64.49.13.27:4963 , piece_id=GRQVBULDQLYHV5ILRPMIWTMWP7DPPSJSKICIWEG4EKUXB7PZ7LSA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182531413 (555h16m25.053614448s) [noise=true , address=145.79.227.43:4160 , piece_id=X42ZZOWQLDUD34IWTT7R66VDHAJGA27IEUTAIMURDF4ELHLIW5PQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182972543 (555h16m25.054055578s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183286231 (555h16m25.054369276s) [noise=true , address=45.91.116.8:4051 , piece_id=AY4G65PDYI34KUS4IAUYIK5WCU3VUMN7XCQJVZTXFXWDMGGYNPBQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183597022 (555h16m25.054680057s) [noise=true , address=77.90.31.21:4895 , piece_id=L5U5SMLBF4R3V7BUZLXRKQ2AEGIQGHPNEJL3FGGLF7DF5V2T4UIA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183869430 (555h16m25.054952475s) [noise=true , address=145.79.227.40:4057 , piece_id=QRTNQJUDFQIOCWSZ7CD5OB5ZE4UJ7W2LKKITFJCS7GJ6G2EIAVWQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.185961945 (555h16m25.05704499s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.186737572 (555h16m25.057820617s) [noise=true , address=77.90.31.13:4630 , piece_id=63OQQKRAHKDKQ7PVN4UAIESK72NW6WOH3Q3QMAVZKSS32PATVKFQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.186816813 (555h16m25.057899848s) [noise=true , address=193.149.164.143:4696 , piece_id=4RTCBEIBUFN72SFD4VQ6UDCBEAU4DRF56HJIZT4L76ETKJUBWTZA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187247843 (555h16m25.058330878s) [noise=true , address=45.91.116.9:4281 , piece_id=AIALW5EEA2VRVELDNWXI64XNNUFQOGZ6424CBLXIOK4AD5S25ZSA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187406526 (555h16m25.058489561s) [noise=true , address=45.91.116.11:4130 , piece_id=ALKG6F653PTF6QYD3NTENX5VR2SE3HYTK4KQN6JF5AI2W5GII3CA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187824422 (555h16m25.058907467s) [noise=true , address=145.79.227.44:4208 , piece_id=2SPAW6GFEDZWNDUCMWO6DNDYFVDGBJOL6XGQEFNTZBZNO7XPKXGQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188212400 (555h16m25.059295445s) [noise=true , address=45.91.116.4:4949 , piece_id=E745VQGK2CVZ2ETETNKIQWDODF4K6U7MDZKGW6SBHQZJEDIV2UXQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188670883 (555h16m25.059753928s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188933434 (555h16m25.060016479s) [noise=true , address=77.90.31.18:4835 , piece_id=OQSXAPKIWHOMPWTP3A2LQ5B7DEI53CGPFLTLPZ4BDDOLZ3VVB5LA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:26.071888944 (555h16m25.942971989s) [noise=true , address=193.149.164.143:4696 , piece_id=4RTCBEIBUFN72SFD4VQ6UDCBEAU4DRF56HJIZT4L76ETKJUBWTZA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: storage node overloaded, request limit: 2000 Dec 28 18:45:26.652316958 (555h16m26.523400013s) [upload=486812]: Next returned nil segment Dec 28 18:45:26.652335714 (555h16m26.523418759s) [upload=486812]: Waiting for error group managing segments... Dec 28 18:45:27.070875407 (555h16m26.941958442s) [noise=true , address=145.79.227.21:4490 , piece_id=CLVXZXJRINCX6ZQP2CNAHLXRAOJI2WZUNAP4LSE2QZ5Y4OJ4AELA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.217018568 (555h16m27.088101603s) [noise=true , address=77.90.31.15:4711 , piece_id=UPAPHGHWJBSBJYIDCN2FI3Z46T2JFUVAQOJPQPGTY4OHIWGG5VPA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.244232238 (555h16m27.115315273s) [noise=true , address=77.90.31.21:4895 , piece_id=L5U5SMLBF4R3V7BUZLXRKQ2AEGIQGHPNEJL3FGGLF7DF5V2T4UIA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.264578651 (555h16m27.135661696s) [noise=true , address=77.90.31.13:4630 , piece_id=63OQQKRAHKDKQ7PVN4UAIESK72NW6WOH3Q3QMAVZKSS32PATVKFQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.277932088 (555h16m27.149015134s) [noise=true , address=45.91.116.9:4281 , piece_id=AIALW5EEA2VRVELDNWXI64XNNUFQOGZ6424CBLXIOK4AD5S25ZSA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.324815542 (555h16m27.195898577s) [noise=true , address=45.91.116.8:4051 , piece_id=AY4G65PDYI34KUS4IAUYIK5WCU3VUMN7XCQJVZTXFXWDMGGYNPBQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.329504170 (555h16m27.200587205s) [noise=true , address=77.90.31.18:4835 , piece_id=OQSXAPKIWHOMPWTP3A2LQ5B7DEI53CGPFLTLPZ4BDDOLZ3VVB5LA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.334083971 (555h16m27.205167006s) [noise=true , address=45.91.116.4:4949 , piece_id=E745VQGK2CVZ2ETETNKIQWDODF4K6U7MDZKGW6SBHQZJEDIV2UXQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.489154753 (555h16m27.360237798s) [noise=true , address=64.49.13.27:4963 , piece_id=GRQVBULDQLYHV5ILRPMIWTMWP7DPPSJSKICIWEG4EKUXB7PZ7LSA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.919436724 (555h16m27.790519759s) [noise=true , address=145.79.227.43:4160 , piece_id=X42ZZOWQLDUD34IWTT7R66VDHAJGA27IEUTAIMURDF4ELHLIW5PQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.987848773 (555h16m27.858931818s) [noise=true , address=145.79.227.30:4768 , piece_id=GPXQOBUIPLRR3MAPDFDZUTL3D35BDNZIG573G4DAUGUH7DJU5T5A, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.988333386 (555h16m27.859416421s) [noise=true , address=145.79.227.44:4208 , piece_id=2SPAW6GFEDZWNDUCMWO6DNDYFVDGBJOL6XGQEFNTZBZNO7XPKXGQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.044342256 (555h16m27.915425291s) [noise=true , address=145.79.227.40:4057 , piece_id=QRTNQJUDFQIOCWSZ7CD5OB5ZE4UJ7W2LKKITFJCS7GJ6G2EIAVWQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.051425193 (555h16m27.922508238s) [noise=true , address=45.91.116.11:4130 , piece_id=ALKG6F653PTF6QYD3NTENX5VR2SE3HYTK4KQN6JF5AI2W5GII3CA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.051449089 (555h16m27.922532144s) [seg_pos={0 0}, upload=486812]: Segment reached optimal threshold of 14 pieces. Dec 28 18:45:28.051675730 (555h16m27.922758775s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051686711 (555h16m27.922769756s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051694917 (555h16m27.922777952s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051704274 (555h16m27.922787309s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051813111 (555h16m27.922896146s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1brZ4RtcK5J3xCPmnbsTccF9hrnj8tqhV1WyMqqrjuitgSGMou): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051818422 (555h16m27.922901457s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051850843 (555h16m27.922933878s) [seg_pos={0 0}, upload=486812]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:45:28.051941646 (555h16m27.923024682s) [upload=486812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:45:28.051946335 (555h16m27.92302937s) [upload=486812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:45:28.356338310 (555h16m28.227421345s) [upload=486812]: Done uploading segments... Dec 28 18:45:28.356361995 (555h16m28.22744504s) [upload=486812]: Upload finished. err: <nil> Duration: 19.430427265s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.X........................X.....................X...] duration: 35.627939711s trace: https://jaeger.dev.storj.io/trace/659b4940275e0f05 FAIL (2025-12-28 18:49:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== begin upload segment. Dec 28 18:45:25.180886421 (555h16m25.051969456s) [noise=true , address=145.79.227.21:4490 , piece_id=CLVXZXJRINCX6ZQP2CNAHLXRAOJI2WZUNAP4LSE2QZ5Y4OJ4AELA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.181484279 (555h16m25.052567314s) [noise=true , address=145.79.227.30:4768 , piece_id=GPXQOBUIPLRR3MAPDFDZUTL3D35BDNZIG573G4DAUGUH7DJU5T5A, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.181903237 (555h16m25.052986282s) [noise=true , address=77.90.31.15:4711 , piece_id=UPAPHGHWJBSBJYIDCN2FI3Z46T2JFUVAQOJPQPGTY4OHIWGG5VPA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182209721 (555h16m25.053292766s) [noise=true , address=64.49.13.27:4963 , piece_id=GRQVBULDQLYHV5ILRPMIWTMWP7DPPSJSKICIWEG4EKUXB7PZ7LSA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182531413 (555h16m25.053614448s) [noise=true , address=145.79.227.43:4160 , piece_id=X42ZZOWQLDUD34IWTT7R66VDHAJGA27IEUTAIMURDF4ELHLIW5PQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182972543 (555h16m25.054055578s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183286231 (555h16m25.054369276s) [noise=true , address=45.91.116.8:4051 , piece_id=AY4G65PDYI34KUS4IAUYIK5WCU3VUMN7XCQJVZTXFXWDMGGYNPBQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183597022 (555h16m25.054680057s) [noise=true , address=77.90.31.21:4895 , piece_id=L5U5SMLBF4R3V7BUZLXRKQ2AEGIQGHPNEJL3FGGLF7DF5V2T4UIA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183869430 (555h16m25.054952475s) [noise=true , address=145.79.227.40:4057 , piece_id=QRTNQJUDFQIOCWSZ7CD5OB5ZE4UJ7W2LKKITFJCS7GJ6G2EIAVWQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.185961945 (555h16m25.05704499s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.186737572 (555h16m25.057820617s) [noise=true , address=77.90.31.13:4630 , piece_id=63OQQKRAHKDKQ7PVN4UAIESK72NW6WOH3Q3QMAVZKSS32PATVKFQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.186816813 (555h16m25.057899848s) [noise=true , address=193.149.164.143:4696 , piece_id=4RTCBEIBUFN72SFD4VQ6UDCBEAU4DRF56HJIZT4L76ETKJUBWTZA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187247843 (555h16m25.058330878s) [noise=true , address=45.91.116.9:4281 , piece_id=AIALW5EEA2VRVELDNWXI64XNNUFQOGZ6424CBLXIOK4AD5S25ZSA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187406526 (555h16m25.058489561s) [noise=true , address=45.91.116.11:4130 , piece_id=ALKG6F653PTF6QYD3NTENX5VR2SE3HYTK4KQN6JF5AI2W5GII3CA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187824422 (555h16m25.058907467s) [noise=true , address=145.79.227.44:4208 , piece_id=2SPAW6GFEDZWNDUCMWO6DNDYFVDGBJOL6XGQEFNTZBZNO7XPKXGQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188212400 (555h16m25.059295445s) [noise=true , address=45.91.116.4:4949 , piece_id=E745VQGK2CVZ2ETETNKIQWDODF4K6U7MDZKGW6SBHQZJEDIV2UXQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188670883 (555h16m25.059753928s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188933434 (555h16m25.060016479s) [noise=true , address=77.90.31.18:4835 , piece_id=OQSXAPKIWHOMPWTP3A2LQ5B7DEI53CGPFLTLPZ4BDDOLZ3VVB5LA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:26.071888944 (555h16m25.942971989s) [noise=true , address=193.149.164.143:4696 , piece_id=4RTCBEIBUFN72SFD4VQ6UDCBEAU4DRF56HJIZT4L76ETKJUBWTZA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: storage node overloaded, request limit: 2000 Dec 28 18:45:26.652316958 (555h16m26.523400013s) [upload=486812]: Next returned nil segment Dec 28 18:45:26.652335714 (555h16m26.523418759s) [upload=486812]: Waiting for error group managing segments... Dec 28 18:45:27.070875407 (555h16m26.941958442s) [noise=true , address=145.79.227.21:4490 , piece_id=CLVXZXJRINCX6ZQP2CNAHLXRAOJI2WZUNAP4LSE2QZ5Y4OJ4AELA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.217018568 (555h16m27.088101603s) [noise=true , address=77.90.31.15:4711 , piece_id=UPAPHGHWJBSBJYIDCN2FI3Z46T2JFUVAQOJPQPGTY4OHIWGG5VPA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.244232238 (555h16m27.115315273s) [noise=true , address=77.90.31.21:4895 , piece_id=L5U5SMLBF4R3V7BUZLXRKQ2AEGIQGHPNEJL3FGGLF7DF5V2T4UIA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.264578651 (555h16m27.135661696s) [noise=true , address=77.90.31.13:4630 , piece_id=63OQQKRAHKDKQ7PVN4UAIESK72NW6WOH3Q3QMAVZKSS32PATVKFQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.277932088 (555h16m27.149015134s) [noise=true , address=45.91.116.9:4281 , piece_id=AIALW5EEA2VRVELDNWXI64XNNUFQOGZ6424CBLXIOK4AD5S25ZSA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.324815542 (555h16m27.195898577s) [noise=true , address=45.91.116.8:4051 , piece_id=AY4G65PDYI34KUS4IAUYIK5WCU3VUMN7XCQJVZTXFXWDMGGYNPBQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.329504170 (555h16m27.200587205s) [noise=true , address=77.90.31.18:4835 , piece_id=OQSXAPKIWHOMPWTP3A2LQ5B7DEI53CGPFLTLPZ4BDDOLZ3VVB5LA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.334083971 (555h16m27.205167006s) [noise=true , address=45.91.116.4:4949 , piece_id=E745VQGK2CVZ2ETETNKIQWDODF4K6U7MDZKGW6SBHQZJEDIV2UXQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.489154753 (555h16m27.360237798s) [noise=true , address=64.49.13.27:4963 , piece_id=GRQVBULDQLYHV5ILRPMIWTMWP7DPPSJSKICIWEG4EKUXB7PZ7LSA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.919436724 (555h16m27.790519759s) [noise=true , address=145.79.227.43:4160 , piece_id=X42ZZOWQLDUD34IWTT7R66VDHAJGA27IEUTAIMURDF4ELHLIW5PQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.987848773 (555h16m27.858931818s) [noise=true , address=145.79.227.30:4768 , piece_id=GPXQOBUIPLRR3MAPDFDZUTL3D35BDNZIG573G4DAUGUH7DJU5T5A, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.988333386 (555h16m27.859416421s) [noise=true , address=145.79.227.44:4208 , piece_id=2SPAW6GFEDZWNDUCMWO6DNDYFVDGBJOL6XGQEFNTZBZNO7XPKXGQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.044342256 (555h16m27.915425291s) [noise=true , address=145.79.227.40:4057 , piece_id=QRTNQJUDFQIOCWSZ7CD5OB5ZE4UJ7W2LKKITFJCS7GJ6G2EIAVWQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.051425193 (555h16m27.922508238s) [noise=true , address=45.91.116.11:4130 , piece_id=ALKG6F653PTF6QYD3NTENX5VR2SE3HYTK4KQN6JF5AI2W5GII3CA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.051449089 (555h16m27.922532144s) [seg_pos={0 0}, upload=486812]: Segment reached optimal threshold of 14 pieces. Dec 28 18:45:28.051675730 (555h16m27.922758775s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051686711 (555h16m27.922769756s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051694917 (555h16m27.922777952s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051704274 (555h16m27.922787309s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051813111 (555h16m27.922896146s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1brZ4RtcK5J3xCPmnbsTccF9hrnj8tqhV1WyMqqrjuitgSGMou): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051818422 (555h16m27.922901457s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051850843 (555h16m27.922933878s) [seg_pos={0 0}, upload=486812]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:45:28.051941646 (555h16m27.923024682s) [upload=486812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:45:28.051946335 (555h16m27.92302937s) [upload=486812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:45:28.356338310 (555h16m28.227421345s) [upload=486812]: Done uploading segments... Dec 28 18:45:28.356361995 (555h16m28.22744504s) [upload=486812]: Upload finished. err: <nil> Duration: 35.627972233s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X....X..X..X....X.X.X.XX..X..X..X...X...XX.......X.X...X..] duration: 6.276957634s trace: https://jaeger.dev.storj.io/trace/1d60d5d8087e6db7 FAIL (2025-12-28 18:48:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A) context deadline exceeded ======LOGS====== 25.850300317 (555h18m25.721383352s) [upload=486840]: Starting upload Dec 28 18:47:25.850388406 (555h18m25.721471441s) [upload=486840]: Done starting upload Dec 28 18:47:25.850510057 (555h18m25.721593092s) [upload=486840]: Uploading segments... Dec 28 18:47:25.850605819 (555h18m25.721688854s) [upload=486840]: Got next segment. Inline: false Dec 28 18:47:25.850634504 (555h18m25.721717539s) [upload=486840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:47:25.850638311 (555h18m25.721721346s) [upload=486840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:47:26.095877088 (555h18m25.966960123s) [seg_pos={0 0}, upload=486840]: Begin upload segment... Dec 28 18:47:26.095932344 (555h18m25.967015389s) [seg_pos={0 0}, upload=486840]: Done begin upload segment. Dec 28 18:47:26.095978802 (555h18m25.967061837s) [noise=true , address=145.79.227.50:4374 , piece_id=AGE3V6L3TNFLVUPLWXVTLKX7A2GDMTKKF6BR5GM4Y2PVLIE7SJPA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096017486 (555h18m25.967100541s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096296628 (555h18m25.967379673s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096691790 (555h18m25.967774835s) [noise=true , address=145.79.227.19:4078 , piece_id=6GMFV2W7SQS4Y44FERO5MZXF6TKEJZHCY3YMSZD367UMXGJR73IQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097015496 (555h18m25.968098531s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097367847 (555h18m25.968450882s) [noise=true , address=145.79.227.45:4231 , piece_id=E2FBWYQWNFJWN6ZVBDT5KJMLJ3JGM735EWVH6FGZRGFOGQ3LXGHA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097651647 (555h18m25.968734682s) [noise=true , address=77.90.31.18:4805 , piece_id=C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098021793 (555h18m25.969104838s) [noise=true , address=145.79.227.12:4767 , piece_id=CZOKOTYGOMGQTFS3RSQRJTHX3DC6IT6LVVFFRVVZ72BK4BUIB3OA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098453214 (555h18m25.969536249s) [noise=true , address=145.79.227.37:4565 , piece_id=BHDEXF2TMQ6TLMEPP6W6CSER5LZTIXUMHSFT3GKKZ3J42XUNLHGQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098930623 (555h18m25.970013668s) [noise=true , address=45.91.116.13:4156 , piece_id=Q36RL5NSW7PSZMPR2GWW7JSL6RCR5VRJBAXT27BEE6W256XDDNNQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.099392252 (555h18m25.970475287s) [noise=true , address=45.91.116.6:4974 , piece_id=63MZ5OJ6BIMZXHFLMCTCNZSSCIZEX7UEXS33WRPJHX4RFBQZLJWQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.100755298 (555h18m25.971838333s) [noise=true , address=77.90.31.16:4736 , piece_id=NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.100928328 (555h18m25.972011373s) [noise=true , address=145.79.227.2:4113 , piece_id=BP4TRNX775AR7IAYCYIEJC6NRNVUWO6J232STFROA6D42U7JI62Q, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.103054576 (555h18m25.974137621s) [noise=true , address=145.79.227.51:4411 , piece_id=NXII74KWEFEFHYHUS6AY2QTKQ3T32AUXLLQI7HUHQY3R7WRLKVXQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.104831500 (555h18m25.975914545s) [noise=true , address=45.91.116.9:4263 , piece_id=7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.106317308 (555h18m25.977400353s) [noise=true , address=145.79.227.11:4730 , piece_id=N4Y5TMAMK4JK33XPETURRJCLTHVKZSHMAYMOQJPM57QWBJAJCGOQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:27.818567118 (555h18m27.689650153s) [upload=486840]: Next returned nil segment Dec 28 18:47:27.818578900 (555h18m27.689661945s) [upload=486840]: Waiting for error group managing segments... Dec 28 18:47:28.101764976 (555h18m27.972848022s) [noise=true , address=145.79.227.50:4374 , piece_id=AGE3V6L3TNFLVUPLWXVTLKX7A2GDMTKKF6BR5GM4Y2PVLIE7SJPA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.102330152 (555h18m27.973413198s) [noise=true , address=145.79.227.45:4231 , piece_id=E2FBWYQWNFJWN6ZVBDT5KJMLJ3JGM735EWVH6FGZRGFOGQ3LXGHA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.104756042 (555h18m27.975839117s) [noise=true , address=145.79.227.51:4411 , piece_id=NXII74KWEFEFHYHUS6AY2QTKQ3T32AUXLLQI7HUHQY3R7WRLKVXQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.105995141 (555h18m27.977078186s) [noise=true , address=145.79.227.19:4078 , piece_id=6GMFV2W7SQS4Y44FERO5MZXF6TKEJZHCY3YMSZD367UMXGJR73IQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.108319868 (555h18m27.979402913s) [noise=true , address=145.79.227.12:4767 , piece_id=CZOKOTYGOMGQTFS3RSQRJTHX3DC6IT6LVVFFRVVZ72BK4BUIB3OA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.119741496 (555h18m27.990824542s) [noise=true , address=145.79.227.37:4565 , piece_id=BHDEXF2TMQ6TLMEPP6W6CSER5LZTIXUMHSFT3GKKZ3J42XUNLHGQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.125589121 (555h18m27.996672156s) [noise=true , address=145.79.227.2:4113 , piece_id=BP4TRNX775AR7IAYCYIEJC6NRNVUWO6J232STFROA6D42U7JI62Q, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.135538287 (555h18m28.006621322s) [noise=true , address=145.79.227.11:4730 , piece_id=N4Y5TMAMK4JK33XPETURRJCLTHVKZSHMAYMOQJPM57QWBJAJCGOQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.317845352 (555h18m28.188928397s) [noise=true , address=77.90.31.16:4736 , piece_id=NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.372520352 (555h18m28.243603387s) [noise=true , address=45.91.116.13:4156 , piece_id=Q36RL5NSW7PSZMPR2GWW7JSL6RCR5VRJBAXT27BEE6W256XDDNNQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.377415213 (555h18m28.248498258s) [noise=true , address=45.91.116.9:4263 , piece_id=7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.384564486 (555h18m28.255647521s) [noise=true , address=77.90.31.18:4805 , piece_id=C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.451238624 (555h18m28.322321659s) [noise=true , address=45.91.116.6:4974 , piece_id=63MZ5OJ6BIMZXHFLMCTCNZSSCIZEX7UEXS33WRPJHX4RFBQZLJWQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.451258964 (555h18m28.322341999s) [seg_pos={0 0}, upload=486840]: Segment reached optimal threshold of 13 pieces. Dec 28 18:47:28.451443845 (555h18m28.32252688s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection Dec 28 18:47:28.451450207 (555h18m28.322533242s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451566598 (555h18m28.322649634s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 18:47:28.451582018 (555h18m28.322665053s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451602367 (555h18m28.322685412s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 28 18:47:28.451608830 (555h18m28.322691865s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451624790 (555h18m28.322707825s) [seg_pos={0 0}, upload=486840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:47:28.451690184 (555h18m28.322773219s) [upload=486840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:47:28.451695344 (555h18m28.322778379s) [upload=486840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:47:28.747747940 (555h18m28.618830975s) [upload=486840]: Done uploading segments... Dec 28 18:47:28.747767095 (555h18m28.618850131s) [upload=486840]: Upload finished. err: <nil> Duration: 6.27698752s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.X....X..X..X....X.X.X.XX..X..X..X...X...XX.......X.X...X.] duration: 5.913408365s trace: https://jaeger.dev.storj.io/trace/1d60d5d8087e6db7 FAIL (2025-12-28 18:48:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A) context deadline exceeded ======LOGS====== 25.850300317 (555h18m25.721383352s) [upload=486840]: Starting upload Dec 28 18:47:25.850388406 (555h18m25.721471441s) [upload=486840]: Done starting upload Dec 28 18:47:25.850510057 (555h18m25.721593092s) [upload=486840]: Uploading segments... Dec 28 18:47:25.850605819 (555h18m25.721688854s) [upload=486840]: Got next segment. Inline: false Dec 28 18:47:25.850634504 (555h18m25.721717539s) [upload=486840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:47:25.850638311 (555h18m25.721721346s) [upload=486840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:47:26.095877088 (555h18m25.966960123s) [seg_pos={0 0}, upload=486840]: Begin upload segment... Dec 28 18:47:26.095932344 (555h18m25.967015389s) [seg_pos={0 0}, upload=486840]: Done begin upload segment. Dec 28 18:47:26.095978802 (555h18m25.967061837s) [noise=true , address=145.79.227.50:4374 , piece_id=AGE3V6L3TNFLVUPLWXVTLKX7A2GDMTKKF6BR5GM4Y2PVLIE7SJPA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096017486 (555h18m25.967100541s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096296628 (555h18m25.967379673s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096691790 (555h18m25.967774835s) [noise=true , address=145.79.227.19:4078 , piece_id=6GMFV2W7SQS4Y44FERO5MZXF6TKEJZHCY3YMSZD367UMXGJR73IQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097015496 (555h18m25.968098531s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097367847 (555h18m25.968450882s) [noise=true , address=145.79.227.45:4231 , piece_id=E2FBWYQWNFJWN6ZVBDT5KJMLJ3JGM735EWVH6FGZRGFOGQ3LXGHA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097651647 (555h18m25.968734682s) [noise=true , address=77.90.31.18:4805 , piece_id=C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098021793 (555h18m25.969104838s) [noise=true , address=145.79.227.12:4767 , piece_id=CZOKOTYGOMGQTFS3RSQRJTHX3DC6IT6LVVFFRVVZ72BK4BUIB3OA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098453214 (555h18m25.969536249s) [noise=true , address=145.79.227.37:4565 , piece_id=BHDEXF2TMQ6TLMEPP6W6CSER5LZTIXUMHSFT3GKKZ3J42XUNLHGQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098930623 (555h18m25.970013668s) [noise=true , address=45.91.116.13:4156 , piece_id=Q36RL5NSW7PSZMPR2GWW7JSL6RCR5VRJBAXT27BEE6W256XDDNNQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.099392252 (555h18m25.970475287s) [noise=true , address=45.91.116.6:4974 , piece_id=63MZ5OJ6BIMZXHFLMCTCNZSSCIZEX7UEXS33WRPJHX4RFBQZLJWQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.100755298 (555h18m25.971838333s) [noise=true , address=77.90.31.16:4736 , piece_id=NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.100928328 (555h18m25.972011373s) [noise=true , address=145.79.227.2:4113 , piece_id=BP4TRNX775AR7IAYCYIEJC6NRNVUWO6J232STFROA6D42U7JI62Q, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.103054576 (555h18m25.974137621s) [noise=true , address=145.79.227.51:4411 , piece_id=NXII74KWEFEFHYHUS6AY2QTKQ3T32AUXLLQI7HUHQY3R7WRLKVXQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.104831500 (555h18m25.975914545s) [noise=true , address=45.91.116.9:4263 , piece_id=7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.106317308 (555h18m25.977400353s) [noise=true , address=145.79.227.11:4730 , piece_id=N4Y5TMAMK4JK33XPETURRJCLTHVKZSHMAYMOQJPM57QWBJAJCGOQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:27.818567118 (555h18m27.689650153s) [upload=486840]: Next returned nil segment Dec 28 18:47:27.818578900 (555h18m27.689661945s) [upload=486840]: Waiting for error group managing segments... Dec 28 18:47:28.101764976 (555h18m27.972848022s) [noise=true , address=145.79.227.50:4374 , piece_id=AGE3V6L3TNFLVUPLWXVTLKX7A2GDMTKKF6BR5GM4Y2PVLIE7SJPA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.102330152 (555h18m27.973413198s) [noise=true , address=145.79.227.45:4231 , piece_id=E2FBWYQWNFJWN6ZVBDT5KJMLJ3JGM735EWVH6FGZRGFOGQ3LXGHA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.104756042 (555h18m27.975839117s) [noise=true , address=145.79.227.51:4411 , piece_id=NXII74KWEFEFHYHUS6AY2QTKQ3T32AUXLLQI7HUHQY3R7WRLKVXQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.105995141 (555h18m27.977078186s) [noise=true , address=145.79.227.19:4078 , piece_id=6GMFV2W7SQS4Y44FERO5MZXF6TKEJZHCY3YMSZD367UMXGJR73IQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.108319868 (555h18m27.979402913s) [noise=true , address=145.79.227.12:4767 , piece_id=CZOKOTYGOMGQTFS3RSQRJTHX3DC6IT6LVVFFRVVZ72BK4BUIB3OA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.119741496 (555h18m27.990824542s) [noise=true , address=145.79.227.37:4565 , piece_id=BHDEXF2TMQ6TLMEPP6W6CSER5LZTIXUMHSFT3GKKZ3J42XUNLHGQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.125589121 (555h18m27.996672156s) [noise=true , address=145.79.227.2:4113 , piece_id=BP4TRNX775AR7IAYCYIEJC6NRNVUWO6J232STFROA6D42U7JI62Q, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.135538287 (555h18m28.006621322s) [noise=true , address=145.79.227.11:4730 , piece_id=N4Y5TMAMK4JK33XPETURRJCLTHVKZSHMAYMOQJPM57QWBJAJCGOQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.317845352 (555h18m28.188928397s) [noise=true , address=77.90.31.16:4736 , piece_id=NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.372520352 (555h18m28.243603387s) [noise=true , address=45.91.116.13:4156 , piece_id=Q36RL5NSW7PSZMPR2GWW7JSL6RCR5VRJBAXT27BEE6W256XDDNNQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.377415213 (555h18m28.248498258s) [noise=true , address=45.91.116.9:4263 , piece_id=7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.384564486 (555h18m28.255647521s) [noise=true , address=77.90.31.18:4805 , piece_id=C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.451238624 (555h18m28.322321659s) [noise=true , address=45.91.116.6:4974 , piece_id=63MZ5OJ6BIMZXHFLMCTCNZSSCIZEX7UEXS33WRPJHX4RFBQZLJWQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.451258964 (555h18m28.322341999s) [seg_pos={0 0}, upload=486840]: Segment reached optimal threshold of 13 pieces. Dec 28 18:47:28.451443845 (555h18m28.32252688s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection Dec 28 18:47:28.451450207 (555h18m28.322533242s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451566598 (555h18m28.322649634s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 18:47:28.451582018 (555h18m28.322665053s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451602367 (555h18m28.322685412s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 28 18:47:28.451608830 (555h18m28.322691865s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451624790 (555h18m28.322707825s) [seg_pos={0 0}, upload=486840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:47:28.451690184 (555h18m28.322773219s) [upload=486840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:47:28.451695344 (555h18m28.322778379s) [upload=486840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:47:28.747747940 (555h18m28.618830975s) [upload=486840]: Done uploading segments... Dec 28 18:47:28.747767095 (555h18m28.618850131s) [upload=486840]: Upload finished. err: <nil> Duration: 5.913424987s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX.......X.X...X] duration: 30.001951956s trace: https://jaeger.dev.storj.io/trace/1d60d5d8087e6db7 FAIL (2025-12-28 18:47:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A) context deadline exceeded ======LOGS====== 25.850300317 (555h18m25.721383352s) [upload=486840]: Starting upload Dec 28 18:47:25.850388406 (555h18m25.721471441s) [upload=486840]: Done starting upload Dec 28 18:47:25.850510057 (555h18m25.721593092s) [upload=486840]: Uploading segments... Dec 28 18:47:25.850605819 (555h18m25.721688854s) [upload=486840]: Got next segment. Inline: false Dec 28 18:47:25.850634504 (555h18m25.721717539s) [upload=486840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:47:25.850638311 (555h18m25.721721346s) [upload=486840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:47:26.095877088 (555h18m25.966960123s) [seg_pos={0 0}, upload=486840]: Begin upload segment... Dec 28 18:47:26.095932344 (555h18m25.967015389s) [seg_pos={0 0}, upload=486840]: Done begin upload segment. Dec 28 18:47:26.095978802 (555h18m25.967061837s) [noise=true , address=145.79.227.50:4374 , piece_id=AGE3V6L3TNFLVUPLWXVTLKX7A2GDMTKKF6BR5GM4Y2PVLIE7SJPA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096017486 (555h18m25.967100541s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096296628 (555h18m25.967379673s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.096691790 (555h18m25.967774835s) [noise=true , address=145.79.227.19:4078 , piece_id=6GMFV2W7SQS4Y44FERO5MZXF6TKEJZHCY3YMSZD367UMXGJR73IQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097015496 (555h18m25.968098531s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097367847 (555h18m25.968450882s) [noise=true , address=145.79.227.45:4231 , piece_id=E2FBWYQWNFJWN6ZVBDT5KJMLJ3JGM735EWVH6FGZRGFOGQ3LXGHA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.097651647 (555h18m25.968734682s) [noise=true , address=77.90.31.18:4805 , piece_id=C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098021793 (555h18m25.969104838s) [noise=true , address=145.79.227.12:4767 , piece_id=CZOKOTYGOMGQTFS3RSQRJTHX3DC6IT6LVVFFRVVZ72BK4BUIB3OA, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098453214 (555h18m25.969536249s) [noise=true , address=145.79.227.37:4565 , piece_id=BHDEXF2TMQ6TLMEPP6W6CSER5LZTIXUMHSFT3GKKZ3J42XUNLHGQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.098930623 (555h18m25.970013668s) [noise=true , address=45.91.116.13:4156 , piece_id=Q36RL5NSW7PSZMPR2GWW7JSL6RCR5VRJBAXT27BEE6W256XDDNNQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.099392252 (555h18m25.970475287s) [noise=true , address=45.91.116.6:4974 , piece_id=63MZ5OJ6BIMZXHFLMCTCNZSSCIZEX7UEXS33WRPJHX4RFBQZLJWQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.100755298 (555h18m25.971838333s) [noise=true , address=77.90.31.16:4736 , piece_id=NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.100928328 (555h18m25.972011373s) [noise=true , address=145.79.227.2:4113 , piece_id=BP4TRNX775AR7IAYCYIEJC6NRNVUWO6J232STFROA6D42U7JI62Q, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.103054576 (555h18m25.974137621s) [noise=true , address=145.79.227.51:4411 , piece_id=NXII74KWEFEFHYHUS6AY2QTKQ3T32AUXLLQI7HUHQY3R7WRLKVXQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.104831500 (555h18m25.975914545s) [noise=true , address=45.91.116.9:4263 , piece_id=7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:26.106317308 (555h18m25.977400353s) [noise=true , address=145.79.227.11:4730 , piece_id=N4Y5TMAMK4JK33XPETURRJCLTHVKZSHMAYMOQJPM57QWBJAJCGOQ, seg_pos={0 0}, upload=486840]: Uploading piece... Dec 28 18:47:27.818567118 (555h18m27.689650153s) [upload=486840]: Next returned nil segment Dec 28 18:47:27.818578900 (555h18m27.689661945s) [upload=486840]: Waiting for error group managing segments... Dec 28 18:47:28.101764976 (555h18m27.972848022s) [noise=true , address=145.79.227.50:4374 , piece_id=AGE3V6L3TNFLVUPLWXVTLKX7A2GDMTKKF6BR5GM4Y2PVLIE7SJPA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.102330152 (555h18m27.973413198s) [noise=true , address=145.79.227.45:4231 , piece_id=E2FBWYQWNFJWN6ZVBDT5KJMLJ3JGM735EWVH6FGZRGFOGQ3LXGHA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.104756042 (555h18m27.975839117s) [noise=true , address=145.79.227.51:4411 , piece_id=NXII74KWEFEFHYHUS6AY2QTKQ3T32AUXLLQI7HUHQY3R7WRLKVXQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.105995141 (555h18m27.977078186s) [noise=true , address=145.79.227.19:4078 , piece_id=6GMFV2W7SQS4Y44FERO5MZXF6TKEJZHCY3YMSZD367UMXGJR73IQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.108319868 (555h18m27.979402913s) [noise=true , address=145.79.227.12:4767 , piece_id=CZOKOTYGOMGQTFS3RSQRJTHX3DC6IT6LVVFFRVVZ72BK4BUIB3OA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.119741496 (555h18m27.990824542s) [noise=true , address=145.79.227.37:4565 , piece_id=BHDEXF2TMQ6TLMEPP6W6CSER5LZTIXUMHSFT3GKKZ3J42XUNLHGQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.125589121 (555h18m27.996672156s) [noise=true , address=145.79.227.2:4113 , piece_id=BP4TRNX775AR7IAYCYIEJC6NRNVUWO6J232STFROA6D42U7JI62Q, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.135538287 (555h18m28.006621322s) [noise=true , address=145.79.227.11:4730 , piece_id=N4Y5TMAMK4JK33XPETURRJCLTHVKZSHMAYMOQJPM57QWBJAJCGOQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.317845352 (555h18m28.188928397s) [noise=true , address=77.90.31.16:4736 , piece_id=NM7QUZ5KZ7QZC7NK23RKDFR3YAIMENSKKWNW4YRE5LTGTDC3H67A, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.372520352 (555h18m28.243603387s) [noise=true , address=45.91.116.13:4156 , piece_id=Q36RL5NSW7PSZMPR2GWW7JSL6RCR5VRJBAXT27BEE6W256XDDNNQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.377415213 (555h18m28.248498258s) [noise=true , address=45.91.116.9:4263 , piece_id=7ZVOBZK2WORVDG64CMSOFVY6Y24N2GXD237B5YT5KCLAPGIEARVQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.384564486 (555h18m28.255647521s) [noise=true , address=77.90.31.18:4805 , piece_id=C4VM5UBDEFWOGOTUN4KZLVTC6Y22OFGLWNFE2OX5UTKCTMTFQJMA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.451238624 (555h18m28.322321659s) [noise=true , address=45.91.116.6:4974 , piece_id=63MZ5OJ6BIMZXHFLMCTCNZSSCIZEX7UEXS33WRPJHX4RFBQZLJWQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: <nil> Dec 28 18:47:28.451258964 (555h18m28.322341999s) [seg_pos={0 0}, upload=486840]: Segment reached optimal threshold of 13 pieces. Dec 28 18:47:28.451443845 (555h18m28.32252688s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55330->145.79.227.17:4971: use of closed network connection Dec 28 18:47:28.451450207 (555h18m28.322533242s) [noise=true , address=145.79.227.17:4971 , piece_id=FMXTXYQN54CFITD7DBFZGUME3SMGLL4S5JF567DHMIRCQJNTJCFQ, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451566598 (555h18m28.322649634s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 18:47:28.451582018 (555h18m28.322665053s) [noise=true , address=45.91.116.10:4082 , piece_id=WUCKTC4DFZ2SWY6EMUTBQESQY7ITTJK4YTFZO7DYJ756WK654VEA, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451602367 (555h18m28.322685412s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 28 18:47:28.451608830 (555h18m28.322691865s) [noise=true , address=145.79.227.33:4870 , piece_id=X3JNRQVTO5EK77BERY7VNMXL6QVQJR3NZWVIWICP7Q3B36UJV2YA, seg_pos={0 0}, upload=486840]: Long tail cancellation due to optimal threshold Dec 28 18:47:28.451624790 (555h18m28.322707825s) [seg_pos={0 0}, upload=486840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:47:28.451690184 (555h18m28.322773219s) [upload=486840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:47:28.451695344 (555h18m28.322778379s) [upload=486840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:47:28.747747940 (555h18m28.618830975s) [upload=486840]: Done uploading segments... Dec 28 18:47:28.747767095 (555h18m28.618850131s) [upload=486840]: Upload finished. err: <nil> Duration: 30.001979128s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.X........................X.....................X..] duration: 27.441646342s trace: https://jaeger.dev.storj.io/trace/659b4940275e0f05 FAIL (2025-12-28 18:47:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== begin upload segment. Dec 28 18:45:25.180886421 (555h16m25.051969456s) [noise=true , address=145.79.227.21:4490 , piece_id=CLVXZXJRINCX6ZQP2CNAHLXRAOJI2WZUNAP4LSE2QZ5Y4OJ4AELA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.181484279 (555h16m25.052567314s) [noise=true , address=145.79.227.30:4768 , piece_id=GPXQOBUIPLRR3MAPDFDZUTL3D35BDNZIG573G4DAUGUH7DJU5T5A, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.181903237 (555h16m25.052986282s) [noise=true , address=77.90.31.15:4711 , piece_id=UPAPHGHWJBSBJYIDCN2FI3Z46T2JFUVAQOJPQPGTY4OHIWGG5VPA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182209721 (555h16m25.053292766s) [noise=true , address=64.49.13.27:4963 , piece_id=GRQVBULDQLYHV5ILRPMIWTMWP7DPPSJSKICIWEG4EKUXB7PZ7LSA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182531413 (555h16m25.053614448s) [noise=true , address=145.79.227.43:4160 , piece_id=X42ZZOWQLDUD34IWTT7R66VDHAJGA27IEUTAIMURDF4ELHLIW5PQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182972543 (555h16m25.054055578s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183286231 (555h16m25.054369276s) [noise=true , address=45.91.116.8:4051 , piece_id=AY4G65PDYI34KUS4IAUYIK5WCU3VUMN7XCQJVZTXFXWDMGGYNPBQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183597022 (555h16m25.054680057s) [noise=true , address=77.90.31.21:4895 , piece_id=L5U5SMLBF4R3V7BUZLXRKQ2AEGIQGHPNEJL3FGGLF7DF5V2T4UIA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183869430 (555h16m25.054952475s) [noise=true , address=145.79.227.40:4057 , piece_id=QRTNQJUDFQIOCWSZ7CD5OB5ZE4UJ7W2LKKITFJCS7GJ6G2EIAVWQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.185961945 (555h16m25.05704499s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.186737572 (555h16m25.057820617s) [noise=true , address=77.90.31.13:4630 , piece_id=63OQQKRAHKDKQ7PVN4UAIESK72NW6WOH3Q3QMAVZKSS32PATVKFQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.186816813 (555h16m25.057899848s) [noise=true , address=193.149.164.143:4696 , piece_id=4RTCBEIBUFN72SFD4VQ6UDCBEAU4DRF56HJIZT4L76ETKJUBWTZA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187247843 (555h16m25.058330878s) [noise=true , address=45.91.116.9:4281 , piece_id=AIALW5EEA2VRVELDNWXI64XNNUFQOGZ6424CBLXIOK4AD5S25ZSA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187406526 (555h16m25.058489561s) [noise=true , address=45.91.116.11:4130 , piece_id=ALKG6F653PTF6QYD3NTENX5VR2SE3HYTK4KQN6JF5AI2W5GII3CA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187824422 (555h16m25.058907467s) [noise=true , address=145.79.227.44:4208 , piece_id=2SPAW6GFEDZWNDUCMWO6DNDYFVDGBJOL6XGQEFNTZBZNO7XPKXGQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188212400 (555h16m25.059295445s) [noise=true , address=45.91.116.4:4949 , piece_id=E745VQGK2CVZ2ETETNKIQWDODF4K6U7MDZKGW6SBHQZJEDIV2UXQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188670883 (555h16m25.059753928s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188933434 (555h16m25.060016479s) [noise=true , address=77.90.31.18:4835 , piece_id=OQSXAPKIWHOMPWTP3A2LQ5B7DEI53CGPFLTLPZ4BDDOLZ3VVB5LA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:26.071888944 (555h16m25.942971989s) [noise=true , address=193.149.164.143:4696 , piece_id=4RTCBEIBUFN72SFD4VQ6UDCBEAU4DRF56HJIZT4L76ETKJUBWTZA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: storage node overloaded, request limit: 2000 Dec 28 18:45:26.652316958 (555h16m26.523400013s) [upload=486812]: Next returned nil segment Dec 28 18:45:26.652335714 (555h16m26.523418759s) [upload=486812]: Waiting for error group managing segments... Dec 28 18:45:27.070875407 (555h16m26.941958442s) [noise=true , address=145.79.227.21:4490 , piece_id=CLVXZXJRINCX6ZQP2CNAHLXRAOJI2WZUNAP4LSE2QZ5Y4OJ4AELA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.217018568 (555h16m27.088101603s) [noise=true , address=77.90.31.15:4711 , piece_id=UPAPHGHWJBSBJYIDCN2FI3Z46T2JFUVAQOJPQPGTY4OHIWGG5VPA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.244232238 (555h16m27.115315273s) [noise=true , address=77.90.31.21:4895 , piece_id=L5U5SMLBF4R3V7BUZLXRKQ2AEGIQGHPNEJL3FGGLF7DF5V2T4UIA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.264578651 (555h16m27.135661696s) [noise=true , address=77.90.31.13:4630 , piece_id=63OQQKRAHKDKQ7PVN4UAIESK72NW6WOH3Q3QMAVZKSS32PATVKFQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.277932088 (555h16m27.149015134s) [noise=true , address=45.91.116.9:4281 , piece_id=AIALW5EEA2VRVELDNWXI64XNNUFQOGZ6424CBLXIOK4AD5S25ZSA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.324815542 (555h16m27.195898577s) [noise=true , address=45.91.116.8:4051 , piece_id=AY4G65PDYI34KUS4IAUYIK5WCU3VUMN7XCQJVZTXFXWDMGGYNPBQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.329504170 (555h16m27.200587205s) [noise=true , address=77.90.31.18:4835 , piece_id=OQSXAPKIWHOMPWTP3A2LQ5B7DEI53CGPFLTLPZ4BDDOLZ3VVB5LA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.334083971 (555h16m27.205167006s) [noise=true , address=45.91.116.4:4949 , piece_id=E745VQGK2CVZ2ETETNKIQWDODF4K6U7MDZKGW6SBHQZJEDIV2UXQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.489154753 (555h16m27.360237798s) [noise=true , address=64.49.13.27:4963 , piece_id=GRQVBULDQLYHV5ILRPMIWTMWP7DPPSJSKICIWEG4EKUXB7PZ7LSA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.919436724 (555h16m27.790519759s) [noise=true , address=145.79.227.43:4160 , piece_id=X42ZZOWQLDUD34IWTT7R66VDHAJGA27IEUTAIMURDF4ELHLIW5PQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.987848773 (555h16m27.858931818s) [noise=true , address=145.79.227.30:4768 , piece_id=GPXQOBUIPLRR3MAPDFDZUTL3D35BDNZIG573G4DAUGUH7DJU5T5A, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.988333386 (555h16m27.859416421s) [noise=true , address=145.79.227.44:4208 , piece_id=2SPAW6GFEDZWNDUCMWO6DNDYFVDGBJOL6XGQEFNTZBZNO7XPKXGQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.044342256 (555h16m27.915425291s) [noise=true , address=145.79.227.40:4057 , piece_id=QRTNQJUDFQIOCWSZ7CD5OB5ZE4UJ7W2LKKITFJCS7GJ6G2EIAVWQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.051425193 (555h16m27.922508238s) [noise=true , address=45.91.116.11:4130 , piece_id=ALKG6F653PTF6QYD3NTENX5VR2SE3HYTK4KQN6JF5AI2W5GII3CA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.051449089 (555h16m27.922532144s) [seg_pos={0 0}, upload=486812]: Segment reached optimal threshold of 14 pieces. Dec 28 18:45:28.051675730 (555h16m27.922758775s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051686711 (555h16m27.922769756s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051694917 (555h16m27.922777952s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051704274 (555h16m27.922787309s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051813111 (555h16m27.922896146s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1brZ4RtcK5J3xCPmnbsTccF9hrnj8tqhV1WyMqqrjuitgSGMou): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051818422 (555h16m27.922901457s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051850843 (555h16m27.922933878s) [seg_pos={0 0}, upload=486812]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:45:28.051941646 (555h16m27.923024682s) [upload=486812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:45:28.051946335 (555h16m27.92302937s) [upload=486812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:45:28.356338310 (555h16m28.227421345s) [upload=486812]: Done uploading segments... Dec 28 18:45:28.356361995 (555h16m28.22744504s) [upload=486812]: Upload finished. err: <nil> Duration: 27.441676369s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX.......X.X...] duration: 6.030569387s trace: https://jaeger.dev.storj.io/trace/3390d4110367484a FAIL (2025-12-28 18:47:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: B434FCNTPVJWLJCNVBAH7PCVNV36VOH7D4LDPRNIJHL2S53VJ2MQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: MTO25OB5QF46F7RKL3VNCRN2FDUDZHXVEZR5WVU7UWRA3DSWCMQA) context deadline exceeded ======LOGS====== Dec 28 18:45:35.906443436 (555h16m35.777526471s) [upload=486813]: Starting upload Dec 28 18:45:35.906512087 (555h16m35.777595132s) [upload=486813]: Done starting upload Dec 28 18:45:35.906542686 (555h16m35.777625721s) [upload=486813]: Uploading segments... Dec 28 18:45:35.906746012 (555h16m35.777829057s) [upload=486813]: Got next segment. Inline: false Dec 28 18:45:35.906782863 (555h16m35.777865908s) [upload=486813]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:45:35.906787622 (555h16m35.777870667s) [upload=486813]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:45:36.160722747 (555h16m36.031805783s) [seg_pos={0 0}, upload=486813]: Begin upload segment... Dec 28 18:45:36.160766101 (555h16m36.031849146s) [seg_pos={0 0}, upload=486813]: Done begin upload segment. Dec 28 18:45:36.160808181 (555h16m36.031891246s) [noise=true , address=145.79.227.15:4873 , piece_id=3M22XHJMBI36BKXJQ7PAZC27OF6B25OUECQOQ3ENLWYXYUY53EHQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.160831936 (555h16m36.031914971s) [noise=true , address=145.79.227.50:4373 , piece_id=WAZLN6D7GJZGEH4X7HPSW464G2MWM6HFEI2BSWRIJPPUUKI5AJQQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.161276512 (555h16m36.032359547s) [noise=true , address=45.91.116.12:4225 , piece_id=GEUBFLBY3AQBZDQTCC2AFH5F7OAPCDUQ5VNLUBFV5TAXQGG5OT5Q, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.161647048 (555h16m36.032730083s) [noise=true , address=145.79.227.42:4121 , piece_id=R3RL7MUEZVKUWKSYUFIBWEIPYT3XX2EFW36FSNE7S3H4EF3WUXYQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.162121642 (555h16m36.033204677s) [noise=true , address=77.90.31.18:4808 , piece_id=B434FCNTPVJWLJCNVBAH7PCVNV36VOH7D4LDPRNIJHL2S53VJ2MQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.163634262 (555h16m36.034717297s) [noise=true , address=145.79.227.40:4051 , piece_id=COGK5Q54NMVRQGERW35NVXEKLR2MFFOSDA7MZ4BGROVC5PBR5OWA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.164506392 (555h16m36.035589438s) [noise=true , address=145.79.227.45:4231 , piece_id=WD7O7FZMM56OR3HNQXVPITQVDVNUKOTAUXQQSH7F2UYQLOMLSKMQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.164777329 (555h16m36.035860374s) [noise=true , address=145.79.227.26:4653 , piece_id=U64SHUAY77USOAV426LGCQEPFCJU7AA27BNFLZ3IZXQ6FQ4ZC3HQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.166056845 (555h16m36.0371399s) [noise=true , address=77.90.31.10:4519 , piece_id=GI4BU3XU4FEUGBIIOFX7XZQZANTZ6EOK4R3FZ2CEEBN4TK2VQPVQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.167280034 (555h16m36.038363069s) [noise=true , address=145.79.227.6:4259 , piece_id=7KFMHQYECMRO7V73XM46L4NWETQTBKAU7TMDGNKAXYY7CAGZISFA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.168392302 (555h16m36.039475347s) [noise=true , address=145.79.227.4:4186 , piece_id=5MVFCT46XABOVBSRGX3WQ2GBL6U6H5XDL6WNDGB3SDTVF3U2DMRA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.169570897 (555h16m36.040653932s) [noise=true , address=145.79.227.36:4517 , piece_id=YCCV4Y62VMPZYNLDNUMHLMCHYEOUBTLBRLE6HLLGZG43AFBIBOMA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.170281860 (555h16m36.041364905s) [noise=true , address=45.91.116.8:4047 , piece_id=MTO25OB5QF46F7RKL3VNCRN2FDUDZHXVEZR5WVU7UWRA3DSWCMQA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.171404928 (555h16m36.042487963s) [noise=true , address=45.91.116.10:4083 , piece_id=KAHNXBSMZZNMZCMDQ4WDIIRKDTUTAK6HHWOWCQBY34GHL2DUCOZQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.172599393 (555h16m36.043682429s) [noise=true , address=145.79.227.48:4303 , piece_id=22GPPEMVPSSUINZLSR34WF63LGYNTHK5ZV7T3S43CWXQKU7GD5RA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.172818030 (555h16m36.043901075s) [noise=true , address=145.79.227.39:4013 , piece_id=6FLMCQDF5RUBHJAGEXP6N6CRBUFUIOQS5EQENGGCWERZ35XJNI6Q, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:37.882334875 (555h16m37.75341792s) [upload=486813]: Next returned nil segment Dec 28 18:45:37.882380301 (555h16m37.753463366s) [upload=486813]: Waiting for error group managing segments... Dec 28 18:45:38.164456322 (555h16m38.035539357s) [noise=true , address=145.79.227.15:4873 , piece_id=3M22XHJMBI36BKXJQ7PAZC27OF6B25OUECQOQ3ENLWYXYUY53EHQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.166882873 (555h16m38.037965908s) [noise=true , address=145.79.227.4:4186 , piece_id=5MVFCT46XABOVBSRGX3WQ2GBL6U6H5XDL6WNDGB3SDTVF3U2DMRA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.169351003 (555h16m38.040434038s) [noise=true , address=145.79.227.6:4259 , piece_id=7KFMHQYECMRO7V73XM46L4NWETQTBKAU7TMDGNKAXYY7CAGZISFA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.170098216 (555h16m38.041181261s) [noise=true , address=145.79.227.36:4517 , piece_id=YCCV4Y62VMPZYNLDNUMHLMCHYEOUBTLBRLE6HLLGZG43AFBIBOMA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.175979766 (555h16m38.047062801s) [noise=true , address=145.79.227.39:4013 , piece_id=6FLMCQDF5RUBHJAGEXP6N6CRBUFUIOQS5EQENGGCWERZ35XJNI6Q, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.176308071 (555h16m38.047391106s) [noise=true , address=145.79.227.26:4653 , piece_id=U64SHUAY77USOAV426LGCQEPFCJU7AA27BNFLZ3IZXQ6FQ4ZC3HQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.183793214 (555h16m38.054876259s) [noise=true , address=145.79.227.42:4121 , piece_id=R3RL7MUEZVKUWKSYUFIBWEIPYT3XX2EFW36FSNE7S3H4EF3WUXYQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.200062653 (555h16m38.071145698s) [noise=true , address=145.79.227.48:4303 , piece_id=22GPPEMVPSSUINZLSR34WF63LGYNTHK5ZV7T3S43CWXQKU7GD5RA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.205211678 (555h16m38.076294723s) [noise=true , address=145.79.227.45:4231 , piece_id=WD7O7FZMM56OR3HNQXVPITQVDVNUKOTAUXQQSH7F2UYQLOMLSKMQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.207043045 (555h16m38.07812609s) [noise=true , address=145.79.227.50:4373 , piece_id=WAZLN6D7GJZGEH4X7HPSW464G2MWM6HFEI2BSWRIJPPUUKI5AJQQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.346274295 (555h16m38.21735734s) [noise=true , address=77.90.31.10:4519 , piece_id=GI4BU3XU4FEUGBIIOFX7XZQZANTZ6EOK4R3FZ2CEEBN4TK2VQPVQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.367812468 (555h16m38.238895503s) [noise=true , address=77.90.31.18:4808 , piece_id=B434FCNTPVJWLJCNVBAH7PCVNV36VOH7D4LDPRNIJHL2S53VJ2MQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.390916743 (555h16m38.261999778s) [noise=true , address=45.91.116.8:4047 , piece_id=MTO25OB5QF46F7RKL3VNCRN2FDUDZHXVEZR5WVU7UWRA3DSWCMQA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.390995512 (555h16m38.262078547s) [seg_pos={0 0}, upload=486813]: Segment reached optimal threshold of 13 pieces. Dec 28 18:45:38.391318237 (555h16m38.262401302s) [noise=true , address=45.91.116.10:4083 , piece_id=KAHNXBSMZZNMZCMDQ4WDIIRKDTUTAK6HHWOWCQBY34GHL2DUCOZQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:38.391327164 (555h16m38.262410199s) [noise=true , address=45.91.116.10:4083 , piece_id=KAHNXBSMZZNMZCMDQ4WDIIRKDTUTAK6HHWOWCQBY34GHL2DUCOZQ, seg_pos={0 0}, upload=486813]: Long tail cancellation due to optimal threshold Dec 28 18:45:38.391538978 (555h16m38.262622023s) [noise=true , address=145.79.227.40:4051 , piece_id=COGK5Q54NMVRQGERW35NVXEKLR2MFFOSDA7MZ4BGROVC5PBR5OWA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:38.391561169 (555h16m38.262644224s) [noise=true , address=145.79.227.40:4051 , piece_id=COGK5Q54NMVRQGERW35NVXEKLR2MFFOSDA7MZ4BGROVC5PBR5OWA, seg_pos={0 0}, upload=486813]: Long tail cancellation due to optimal threshold Dec 28 18:45:38.391590916 (555h16m38.262673961s) [noise=true , address=45.91.116.12:4225 , piece_id=GEUBFLBY3AQBZDQTCC2AFH5F7OAPCDUQ5VNLUBFV5TAXQGG5OT5Q, seg_pos={0 0}, upload=486813]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:38.391600154 (555h16m38.262683199s) [noise=true , address=45.91.116.12:4225 , piece_id=GEUBFLBY3AQBZDQTCC2AFH5F7OAPCDUQ5VNLUBFV5TAXQGG5OT5Q, seg_pos={0 0}, upload=486813]: Long tail cancellation due to optimal threshold Dec 28 18:45:38.391625311 (555h16m38.262708357s) [seg_pos={0 0}, upload=486813]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:45:38.391707178 (555h16m38.262790213s) [upload=486813]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:45:38.391712017 (555h16m38.262795052s) [upload=486813]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:45:38.691244197 (555h16m38.562327233s) [upload=486813]: Done uploading segments... Dec 28 18:45:38.691269095 (555h16m38.56235214s) [upload=486813]: Upload finished. err: <nil> Duration: 6.030594946s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.X........................X.....................X.] duration: 25.921187233s trace: https://jaeger.dev.storj.io/trace/659b4940275e0f05 FAIL (2025-12-28 18:46:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== begin upload segment. Dec 28 18:45:25.180886421 (555h16m25.051969456s) [noise=true , address=145.79.227.21:4490 , piece_id=CLVXZXJRINCX6ZQP2CNAHLXRAOJI2WZUNAP4LSE2QZ5Y4OJ4AELA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.181484279 (555h16m25.052567314s) [noise=true , address=145.79.227.30:4768 , piece_id=GPXQOBUIPLRR3MAPDFDZUTL3D35BDNZIG573G4DAUGUH7DJU5T5A, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.181903237 (555h16m25.052986282s) [noise=true , address=77.90.31.15:4711 , piece_id=UPAPHGHWJBSBJYIDCN2FI3Z46T2JFUVAQOJPQPGTY4OHIWGG5VPA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182209721 (555h16m25.053292766s) [noise=true , address=64.49.13.27:4963 , piece_id=GRQVBULDQLYHV5ILRPMIWTMWP7DPPSJSKICIWEG4EKUXB7PZ7LSA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182531413 (555h16m25.053614448s) [noise=true , address=145.79.227.43:4160 , piece_id=X42ZZOWQLDUD34IWTT7R66VDHAJGA27IEUTAIMURDF4ELHLIW5PQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182972543 (555h16m25.054055578s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183286231 (555h16m25.054369276s) [noise=true , address=45.91.116.8:4051 , piece_id=AY4G65PDYI34KUS4IAUYIK5WCU3VUMN7XCQJVZTXFXWDMGGYNPBQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183597022 (555h16m25.054680057s) [noise=true , address=77.90.31.21:4895 , piece_id=L5U5SMLBF4R3V7BUZLXRKQ2AEGIQGHPNEJL3FGGLF7DF5V2T4UIA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183869430 (555h16m25.054952475s) [noise=true , address=145.79.227.40:4057 , piece_id=QRTNQJUDFQIOCWSZ7CD5OB5ZE4UJ7W2LKKITFJCS7GJ6G2EIAVWQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.185961945 (555h16m25.05704499s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.186737572 (555h16m25.057820617s) [noise=true , address=77.90.31.13:4630 , piece_id=63OQQKRAHKDKQ7PVN4UAIESK72NW6WOH3Q3QMAVZKSS32PATVKFQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.186816813 (555h16m25.057899848s) [noise=true , address=193.149.164.143:4696 , piece_id=4RTCBEIBUFN72SFD4VQ6UDCBEAU4DRF56HJIZT4L76ETKJUBWTZA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187247843 (555h16m25.058330878s) [noise=true , address=45.91.116.9:4281 , piece_id=AIALW5EEA2VRVELDNWXI64XNNUFQOGZ6424CBLXIOK4AD5S25ZSA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187406526 (555h16m25.058489561s) [noise=true , address=45.91.116.11:4130 , piece_id=ALKG6F653PTF6QYD3NTENX5VR2SE3HYTK4KQN6JF5AI2W5GII3CA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187824422 (555h16m25.058907467s) [noise=true , address=145.79.227.44:4208 , piece_id=2SPAW6GFEDZWNDUCMWO6DNDYFVDGBJOL6XGQEFNTZBZNO7XPKXGQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188212400 (555h16m25.059295445s) [noise=true , address=45.91.116.4:4949 , piece_id=E745VQGK2CVZ2ETETNKIQWDODF4K6U7MDZKGW6SBHQZJEDIV2UXQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188670883 (555h16m25.059753928s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188933434 (555h16m25.060016479s) [noise=true , address=77.90.31.18:4835 , piece_id=OQSXAPKIWHOMPWTP3A2LQ5B7DEI53CGPFLTLPZ4BDDOLZ3VVB5LA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:26.071888944 (555h16m25.942971989s) [noise=true , address=193.149.164.143:4696 , piece_id=4RTCBEIBUFN72SFD4VQ6UDCBEAU4DRF56HJIZT4L76ETKJUBWTZA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: storage node overloaded, request limit: 2000 Dec 28 18:45:26.652316958 (555h16m26.523400013s) [upload=486812]: Next returned nil segment Dec 28 18:45:26.652335714 (555h16m26.523418759s) [upload=486812]: Waiting for error group managing segments... Dec 28 18:45:27.070875407 (555h16m26.941958442s) [noise=true , address=145.79.227.21:4490 , piece_id=CLVXZXJRINCX6ZQP2CNAHLXRAOJI2WZUNAP4LSE2QZ5Y4OJ4AELA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.217018568 (555h16m27.088101603s) [noise=true , address=77.90.31.15:4711 , piece_id=UPAPHGHWJBSBJYIDCN2FI3Z46T2JFUVAQOJPQPGTY4OHIWGG5VPA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.244232238 (555h16m27.115315273s) [noise=true , address=77.90.31.21:4895 , piece_id=L5U5SMLBF4R3V7BUZLXRKQ2AEGIQGHPNEJL3FGGLF7DF5V2T4UIA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.264578651 (555h16m27.135661696s) [noise=true , address=77.90.31.13:4630 , piece_id=63OQQKRAHKDKQ7PVN4UAIESK72NW6WOH3Q3QMAVZKSS32PATVKFQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.277932088 (555h16m27.149015134s) [noise=true , address=45.91.116.9:4281 , piece_id=AIALW5EEA2VRVELDNWXI64XNNUFQOGZ6424CBLXIOK4AD5S25ZSA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.324815542 (555h16m27.195898577s) [noise=true , address=45.91.116.8:4051 , piece_id=AY4G65PDYI34KUS4IAUYIK5WCU3VUMN7XCQJVZTXFXWDMGGYNPBQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.329504170 (555h16m27.200587205s) [noise=true , address=77.90.31.18:4835 , piece_id=OQSXAPKIWHOMPWTP3A2LQ5B7DEI53CGPFLTLPZ4BDDOLZ3VVB5LA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.334083971 (555h16m27.205167006s) [noise=true , address=45.91.116.4:4949 , piece_id=E745VQGK2CVZ2ETETNKIQWDODF4K6U7MDZKGW6SBHQZJEDIV2UXQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.489154753 (555h16m27.360237798s) [noise=true , address=64.49.13.27:4963 , piece_id=GRQVBULDQLYHV5ILRPMIWTMWP7DPPSJSKICIWEG4EKUXB7PZ7LSA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.919436724 (555h16m27.790519759s) [noise=true , address=145.79.227.43:4160 , piece_id=X42ZZOWQLDUD34IWTT7R66VDHAJGA27IEUTAIMURDF4ELHLIW5PQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.987848773 (555h16m27.858931818s) [noise=true , address=145.79.227.30:4768 , piece_id=GPXQOBUIPLRR3MAPDFDZUTL3D35BDNZIG573G4DAUGUH7DJU5T5A, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.988333386 (555h16m27.859416421s) [noise=true , address=145.79.227.44:4208 , piece_id=2SPAW6GFEDZWNDUCMWO6DNDYFVDGBJOL6XGQEFNTZBZNO7XPKXGQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.044342256 (555h16m27.915425291s) [noise=true , address=145.79.227.40:4057 , piece_id=QRTNQJUDFQIOCWSZ7CD5OB5ZE4UJ7W2LKKITFJCS7GJ6G2EIAVWQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.051425193 (555h16m27.922508238s) [noise=true , address=45.91.116.11:4130 , piece_id=ALKG6F653PTF6QYD3NTENX5VR2SE3HYTK4KQN6JF5AI2W5GII3CA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.051449089 (555h16m27.922532144s) [seg_pos={0 0}, upload=486812]: Segment reached optimal threshold of 14 pieces. Dec 28 18:45:28.051675730 (555h16m27.922758775s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051686711 (555h16m27.922769756s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051694917 (555h16m27.922777952s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051704274 (555h16m27.922787309s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051813111 (555h16m27.922896146s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1brZ4RtcK5J3xCPmnbsTccF9hrnj8tqhV1WyMqqrjuitgSGMou): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051818422 (555h16m27.922901457s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051850843 (555h16m27.922933878s) [seg_pos={0 0}, upload=486812]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:45:28.051941646 (555h16m27.923024682s) [upload=486812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:45:28.051946335 (555h16m27.92302937s) [upload=486812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:45:28.356338310 (555h16m28.227421345s) [upload=486812]: Done uploading segments... Dec 28 18:45:28.356361995 (555h16m28.22744504s) [upload=486812]: Upload finished. err: <nil> Duration: 25.921209125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX.......X.X..] duration: 6.108304468s trace: https://jaeger.dev.storj.io/trace/3390d4110367484a FAIL (2025-12-28 18:46:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: B434FCNTPVJWLJCNVBAH7PCVNV36VOH7D4LDPRNIJHL2S53VJ2MQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: MTO25OB5QF46F7RKL3VNCRN2FDUDZHXVEZR5WVU7UWRA3DSWCMQA) context deadline exceeded ======LOGS====== Dec 28 18:45:35.906443436 (555h16m35.777526471s) [upload=486813]: Starting upload Dec 28 18:45:35.906512087 (555h16m35.777595132s) [upload=486813]: Done starting upload Dec 28 18:45:35.906542686 (555h16m35.777625721s) [upload=486813]: Uploading segments... Dec 28 18:45:35.906746012 (555h16m35.777829057s) [upload=486813]: Got next segment. Inline: false Dec 28 18:45:35.906782863 (555h16m35.777865908s) [upload=486813]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:45:35.906787622 (555h16m35.777870667s) [upload=486813]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:45:36.160722747 (555h16m36.031805783s) [seg_pos={0 0}, upload=486813]: Begin upload segment... Dec 28 18:45:36.160766101 (555h16m36.031849146s) [seg_pos={0 0}, upload=486813]: Done begin upload segment. Dec 28 18:45:36.160808181 (555h16m36.031891246s) [noise=true , address=145.79.227.15:4873 , piece_id=3M22XHJMBI36BKXJQ7PAZC27OF6B25OUECQOQ3ENLWYXYUY53EHQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.160831936 (555h16m36.031914971s) [noise=true , address=145.79.227.50:4373 , piece_id=WAZLN6D7GJZGEH4X7HPSW464G2MWM6HFEI2BSWRIJPPUUKI5AJQQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.161276512 (555h16m36.032359547s) [noise=true , address=45.91.116.12:4225 , piece_id=GEUBFLBY3AQBZDQTCC2AFH5F7OAPCDUQ5VNLUBFV5TAXQGG5OT5Q, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.161647048 (555h16m36.032730083s) [noise=true , address=145.79.227.42:4121 , piece_id=R3RL7MUEZVKUWKSYUFIBWEIPYT3XX2EFW36FSNE7S3H4EF3WUXYQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.162121642 (555h16m36.033204677s) [noise=true , address=77.90.31.18:4808 , piece_id=B434FCNTPVJWLJCNVBAH7PCVNV36VOH7D4LDPRNIJHL2S53VJ2MQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.163634262 (555h16m36.034717297s) [noise=true , address=145.79.227.40:4051 , piece_id=COGK5Q54NMVRQGERW35NVXEKLR2MFFOSDA7MZ4BGROVC5PBR5OWA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.164506392 (555h16m36.035589438s) [noise=true , address=145.79.227.45:4231 , piece_id=WD7O7FZMM56OR3HNQXVPITQVDVNUKOTAUXQQSH7F2UYQLOMLSKMQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.164777329 (555h16m36.035860374s) [noise=true , address=145.79.227.26:4653 , piece_id=U64SHUAY77USOAV426LGCQEPFCJU7AA27BNFLZ3IZXQ6FQ4ZC3HQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.166056845 (555h16m36.0371399s) [noise=true , address=77.90.31.10:4519 , piece_id=GI4BU3XU4FEUGBIIOFX7XZQZANTZ6EOK4R3FZ2CEEBN4TK2VQPVQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.167280034 (555h16m36.038363069s) [noise=true , address=145.79.227.6:4259 , piece_id=7KFMHQYECMRO7V73XM46L4NWETQTBKAU7TMDGNKAXYY7CAGZISFA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.168392302 (555h16m36.039475347s) [noise=true , address=145.79.227.4:4186 , piece_id=5MVFCT46XABOVBSRGX3WQ2GBL6U6H5XDL6WNDGB3SDTVF3U2DMRA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.169570897 (555h16m36.040653932s) [noise=true , address=145.79.227.36:4517 , piece_id=YCCV4Y62VMPZYNLDNUMHLMCHYEOUBTLBRLE6HLLGZG43AFBIBOMA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.170281860 (555h16m36.041364905s) [noise=true , address=45.91.116.8:4047 , piece_id=MTO25OB5QF46F7RKL3VNCRN2FDUDZHXVEZR5WVU7UWRA3DSWCMQA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.171404928 (555h16m36.042487963s) [noise=true , address=45.91.116.10:4083 , piece_id=KAHNXBSMZZNMZCMDQ4WDIIRKDTUTAK6HHWOWCQBY34GHL2DUCOZQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.172599393 (555h16m36.043682429s) [noise=true , address=145.79.227.48:4303 , piece_id=22GPPEMVPSSUINZLSR34WF63LGYNTHK5ZV7T3S43CWXQKU7GD5RA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.172818030 (555h16m36.043901075s) [noise=true , address=145.79.227.39:4013 , piece_id=6FLMCQDF5RUBHJAGEXP6N6CRBUFUIOQS5EQENGGCWERZ35XJNI6Q, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:37.882334875 (555h16m37.75341792s) [upload=486813]: Next returned nil segment Dec 28 18:45:37.882380301 (555h16m37.753463366s) [upload=486813]: Waiting for error group managing segments... Dec 28 18:45:38.164456322 (555h16m38.035539357s) [noise=true , address=145.79.227.15:4873 , piece_id=3M22XHJMBI36BKXJQ7PAZC27OF6B25OUECQOQ3ENLWYXYUY53EHQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.166882873 (555h16m38.037965908s) [noise=true , address=145.79.227.4:4186 , piece_id=5MVFCT46XABOVBSRGX3WQ2GBL6U6H5XDL6WNDGB3SDTVF3U2DMRA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.169351003 (555h16m38.040434038s) [noise=true , address=145.79.227.6:4259 , piece_id=7KFMHQYECMRO7V73XM46L4NWETQTBKAU7TMDGNKAXYY7CAGZISFA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.170098216 (555h16m38.041181261s) [noise=true , address=145.79.227.36:4517 , piece_id=YCCV4Y62VMPZYNLDNUMHLMCHYEOUBTLBRLE6HLLGZG43AFBIBOMA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.175979766 (555h16m38.047062801s) [noise=true , address=145.79.227.39:4013 , piece_id=6FLMCQDF5RUBHJAGEXP6N6CRBUFUIOQS5EQENGGCWERZ35XJNI6Q, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.176308071 (555h16m38.047391106s) [noise=true , address=145.79.227.26:4653 , piece_id=U64SHUAY77USOAV426LGCQEPFCJU7AA27BNFLZ3IZXQ6FQ4ZC3HQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.183793214 (555h16m38.054876259s) [noise=true , address=145.79.227.42:4121 , piece_id=R3RL7MUEZVKUWKSYUFIBWEIPYT3XX2EFW36FSNE7S3H4EF3WUXYQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.200062653 (555h16m38.071145698s) [noise=true , address=145.79.227.48:4303 , piece_id=22GPPEMVPSSUINZLSR34WF63LGYNTHK5ZV7T3S43CWXQKU7GD5RA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.205211678 (555h16m38.076294723s) [noise=true , address=145.79.227.45:4231 , piece_id=WD7O7FZMM56OR3HNQXVPITQVDVNUKOTAUXQQSH7F2UYQLOMLSKMQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.207043045 (555h16m38.07812609s) [noise=true , address=145.79.227.50:4373 , piece_id=WAZLN6D7GJZGEH4X7HPSW464G2MWM6HFEI2BSWRIJPPUUKI5AJQQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.346274295 (555h16m38.21735734s) [noise=true , address=77.90.31.10:4519 , piece_id=GI4BU3XU4FEUGBIIOFX7XZQZANTZ6EOK4R3FZ2CEEBN4TK2VQPVQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.367812468 (555h16m38.238895503s) [noise=true , address=77.90.31.18:4808 , piece_id=B434FCNTPVJWLJCNVBAH7PCVNV36VOH7D4LDPRNIJHL2S53VJ2MQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.390916743 (555h16m38.261999778s) [noise=true , address=45.91.116.8:4047 , piece_id=MTO25OB5QF46F7RKL3VNCRN2FDUDZHXVEZR5WVU7UWRA3DSWCMQA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.390995512 (555h16m38.262078547s) [seg_pos={0 0}, upload=486813]: Segment reached optimal threshold of 13 pieces. Dec 28 18:45:38.391318237 (555h16m38.262401302s) [noise=true , address=45.91.116.10:4083 , piece_id=KAHNXBSMZZNMZCMDQ4WDIIRKDTUTAK6HHWOWCQBY34GHL2DUCOZQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:38.391327164 (555h16m38.262410199s) [noise=true , address=45.91.116.10:4083 , piece_id=KAHNXBSMZZNMZCMDQ4WDIIRKDTUTAK6HHWOWCQBY34GHL2DUCOZQ, seg_pos={0 0}, upload=486813]: Long tail cancellation due to optimal threshold Dec 28 18:45:38.391538978 (555h16m38.262622023s) [noise=true , address=145.79.227.40:4051 , piece_id=COGK5Q54NMVRQGERW35NVXEKLR2MFFOSDA7MZ4BGROVC5PBR5OWA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:38.391561169 (555h16m38.262644224s) [noise=true , address=145.79.227.40:4051 , piece_id=COGK5Q54NMVRQGERW35NVXEKLR2MFFOSDA7MZ4BGROVC5PBR5OWA, seg_pos={0 0}, upload=486813]: Long tail cancellation due to optimal threshold Dec 28 18:45:38.391590916 (555h16m38.262673961s) [noise=true , address=45.91.116.12:4225 , piece_id=GEUBFLBY3AQBZDQTCC2AFH5F7OAPCDUQ5VNLUBFV5TAXQGG5OT5Q, seg_pos={0 0}, upload=486813]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:38.391600154 (555h16m38.262683199s) [noise=true , address=45.91.116.12:4225 , piece_id=GEUBFLBY3AQBZDQTCC2AFH5F7OAPCDUQ5VNLUBFV5TAXQGG5OT5Q, seg_pos={0 0}, upload=486813]: Long tail cancellation due to optimal threshold Dec 28 18:45:38.391625311 (555h16m38.262708357s) [seg_pos={0 0}, upload=486813]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:45:38.391707178 (555h16m38.262790213s) [upload=486813]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:45:38.391712017 (555h16m38.262795052s) [upload=486813]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:45:38.691244197 (555h16m38.562327233s) [upload=486813]: Done uploading segments... Dec 28 18:45:38.691269095 (555h16m38.56235214s) [upload=486813]: Upload finished. err: <nil> Duration: 6.108337101s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX.......X.X.] duration: 17.723286022s trace: https://jaeger.dev.storj.io/trace/3390d4110367484a FAIL (2025-12-28 18:46:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: B434FCNTPVJWLJCNVBAH7PCVNV36VOH7D4LDPRNIJHL2S53VJ2MQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: MTO25OB5QF46F7RKL3VNCRN2FDUDZHXVEZR5WVU7UWRA3DSWCMQA) context deadline exceeded ======LOGS====== Dec 28 18:45:35.906443436 (555h16m35.777526471s) [upload=486813]: Starting upload Dec 28 18:45:35.906512087 (555h16m35.777595132s) [upload=486813]: Done starting upload Dec 28 18:45:35.906542686 (555h16m35.777625721s) [upload=486813]: Uploading segments... Dec 28 18:45:35.906746012 (555h16m35.777829057s) [upload=486813]: Got next segment. Inline: false Dec 28 18:45:35.906782863 (555h16m35.777865908s) [upload=486813]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:45:35.906787622 (555h16m35.777870667s) [upload=486813]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:45:36.160722747 (555h16m36.031805783s) [seg_pos={0 0}, upload=486813]: Begin upload segment... Dec 28 18:45:36.160766101 (555h16m36.031849146s) [seg_pos={0 0}, upload=486813]: Done begin upload segment. Dec 28 18:45:36.160808181 (555h16m36.031891246s) [noise=true , address=145.79.227.15:4873 , piece_id=3M22XHJMBI36BKXJQ7PAZC27OF6B25OUECQOQ3ENLWYXYUY53EHQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.160831936 (555h16m36.031914971s) [noise=true , address=145.79.227.50:4373 , piece_id=WAZLN6D7GJZGEH4X7HPSW464G2MWM6HFEI2BSWRIJPPUUKI5AJQQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.161276512 (555h16m36.032359547s) [noise=true , address=45.91.116.12:4225 , piece_id=GEUBFLBY3AQBZDQTCC2AFH5F7OAPCDUQ5VNLUBFV5TAXQGG5OT5Q, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.161647048 (555h16m36.032730083s) [noise=true , address=145.79.227.42:4121 , piece_id=R3RL7MUEZVKUWKSYUFIBWEIPYT3XX2EFW36FSNE7S3H4EF3WUXYQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.162121642 (555h16m36.033204677s) [noise=true , address=77.90.31.18:4808 , piece_id=B434FCNTPVJWLJCNVBAH7PCVNV36VOH7D4LDPRNIJHL2S53VJ2MQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.163634262 (555h16m36.034717297s) [noise=true , address=145.79.227.40:4051 , piece_id=COGK5Q54NMVRQGERW35NVXEKLR2MFFOSDA7MZ4BGROVC5PBR5OWA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.164506392 (555h16m36.035589438s) [noise=true , address=145.79.227.45:4231 , piece_id=WD7O7FZMM56OR3HNQXVPITQVDVNUKOTAUXQQSH7F2UYQLOMLSKMQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.164777329 (555h16m36.035860374s) [noise=true , address=145.79.227.26:4653 , piece_id=U64SHUAY77USOAV426LGCQEPFCJU7AA27BNFLZ3IZXQ6FQ4ZC3HQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.166056845 (555h16m36.0371399s) [noise=true , address=77.90.31.10:4519 , piece_id=GI4BU3XU4FEUGBIIOFX7XZQZANTZ6EOK4R3FZ2CEEBN4TK2VQPVQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.167280034 (555h16m36.038363069s) [noise=true , address=145.79.227.6:4259 , piece_id=7KFMHQYECMRO7V73XM46L4NWETQTBKAU7TMDGNKAXYY7CAGZISFA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.168392302 (555h16m36.039475347s) [noise=true , address=145.79.227.4:4186 , piece_id=5MVFCT46XABOVBSRGX3WQ2GBL6U6H5XDL6WNDGB3SDTVF3U2DMRA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.169570897 (555h16m36.040653932s) [noise=true , address=145.79.227.36:4517 , piece_id=YCCV4Y62VMPZYNLDNUMHLMCHYEOUBTLBRLE6HLLGZG43AFBIBOMA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.170281860 (555h16m36.041364905s) [noise=true , address=45.91.116.8:4047 , piece_id=MTO25OB5QF46F7RKL3VNCRN2FDUDZHXVEZR5WVU7UWRA3DSWCMQA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.171404928 (555h16m36.042487963s) [noise=true , address=45.91.116.10:4083 , piece_id=KAHNXBSMZZNMZCMDQ4WDIIRKDTUTAK6HHWOWCQBY34GHL2DUCOZQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.172599393 (555h16m36.043682429s) [noise=true , address=145.79.227.48:4303 , piece_id=22GPPEMVPSSUINZLSR34WF63LGYNTHK5ZV7T3S43CWXQKU7GD5RA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.172818030 (555h16m36.043901075s) [noise=true , address=145.79.227.39:4013 , piece_id=6FLMCQDF5RUBHJAGEXP6N6CRBUFUIOQS5EQENGGCWERZ35XJNI6Q, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:37.882334875 (555h16m37.75341792s) [upload=486813]: Next returned nil segment Dec 28 18:45:37.882380301 (555h16m37.753463366s) [upload=486813]: Waiting for error group managing segments... Dec 28 18:45:38.164456322 (555h16m38.035539357s) [noise=true , address=145.79.227.15:4873 , piece_id=3M22XHJMBI36BKXJQ7PAZC27OF6B25OUECQOQ3ENLWYXYUY53EHQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.166882873 (555h16m38.037965908s) [noise=true , address=145.79.227.4:4186 , piece_id=5MVFCT46XABOVBSRGX3WQ2GBL6U6H5XDL6WNDGB3SDTVF3U2DMRA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.169351003 (555h16m38.040434038s) [noise=true , address=145.79.227.6:4259 , piece_id=7KFMHQYECMRO7V73XM46L4NWETQTBKAU7TMDGNKAXYY7CAGZISFA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.170098216 (555h16m38.041181261s) [noise=true , address=145.79.227.36:4517 , piece_id=YCCV4Y62VMPZYNLDNUMHLMCHYEOUBTLBRLE6HLLGZG43AFBIBOMA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.175979766 (555h16m38.047062801s) [noise=true , address=145.79.227.39:4013 , piece_id=6FLMCQDF5RUBHJAGEXP6N6CRBUFUIOQS5EQENGGCWERZ35XJNI6Q, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.176308071 (555h16m38.047391106s) [noise=true , address=145.79.227.26:4653 , piece_id=U64SHUAY77USOAV426LGCQEPFCJU7AA27BNFLZ3IZXQ6FQ4ZC3HQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.183793214 (555h16m38.054876259s) [noise=true , address=145.79.227.42:4121 , piece_id=R3RL7MUEZVKUWKSYUFIBWEIPYT3XX2EFW36FSNE7S3H4EF3WUXYQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.200062653 (555h16m38.071145698s) [noise=true , address=145.79.227.48:4303 , piece_id=22GPPEMVPSSUINZLSR34WF63LGYNTHK5ZV7T3S43CWXQKU7GD5RA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.205211678 (555h16m38.076294723s) [noise=true , address=145.79.227.45:4231 , piece_id=WD7O7FZMM56OR3HNQXVPITQVDVNUKOTAUXQQSH7F2UYQLOMLSKMQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.207043045 (555h16m38.07812609s) [noise=true , address=145.79.227.50:4373 , piece_id=WAZLN6D7GJZGEH4X7HPSW464G2MWM6HFEI2BSWRIJPPUUKI5AJQQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.346274295 (555h16m38.21735734s) [noise=true , address=77.90.31.10:4519 , piece_id=GI4BU3XU4FEUGBIIOFX7XZQZANTZ6EOK4R3FZ2CEEBN4TK2VQPVQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.367812468 (555h16m38.238895503s) [noise=true , address=77.90.31.18:4808 , piece_id=B434FCNTPVJWLJCNVBAH7PCVNV36VOH7D4LDPRNIJHL2S53VJ2MQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.390916743 (555h16m38.261999778s) [noise=true , address=45.91.116.8:4047 , piece_id=MTO25OB5QF46F7RKL3VNCRN2FDUDZHXVEZR5WVU7UWRA3DSWCMQA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.390995512 (555h16m38.262078547s) [seg_pos={0 0}, upload=486813]: Segment reached optimal threshold of 13 pieces. Dec 28 18:45:38.391318237 (555h16m38.262401302s) [noise=true , address=45.91.116.10:4083 , piece_id=KAHNXBSMZZNMZCMDQ4WDIIRKDTUTAK6HHWOWCQBY34GHL2DUCOZQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:38.391327164 (555h16m38.262410199s) [noise=true , address=45.91.116.10:4083 , piece_id=KAHNXBSMZZNMZCMDQ4WDIIRKDTUTAK6HHWOWCQBY34GHL2DUCOZQ, seg_pos={0 0}, upload=486813]: Long tail cancellation due to optimal threshold Dec 28 18:45:38.391538978 (555h16m38.262622023s) [noise=true , address=145.79.227.40:4051 , piece_id=COGK5Q54NMVRQGERW35NVXEKLR2MFFOSDA7MZ4BGROVC5PBR5OWA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:38.391561169 (555h16m38.262644224s) [noise=true , address=145.79.227.40:4051 , piece_id=COGK5Q54NMVRQGERW35NVXEKLR2MFFOSDA7MZ4BGROVC5PBR5OWA, seg_pos={0 0}, upload=486813]: Long tail cancellation due to optimal threshold Dec 28 18:45:38.391590916 (555h16m38.262673961s) [noise=true , address=45.91.116.12:4225 , piece_id=GEUBFLBY3AQBZDQTCC2AFH5F7OAPCDUQ5VNLUBFV5TAXQGG5OT5Q, seg_pos={0 0}, upload=486813]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:38.391600154 (555h16m38.262683199s) [noise=true , address=45.91.116.12:4225 , piece_id=GEUBFLBY3AQBZDQTCC2AFH5F7OAPCDUQ5VNLUBFV5TAXQGG5OT5Q, seg_pos={0 0}, upload=486813]: Long tail cancellation due to optimal threshold Dec 28 18:45:38.391625311 (555h16m38.262708357s) [seg_pos={0 0}, upload=486813]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:45:38.391707178 (555h16m38.262790213s) [upload=486813]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:45:38.391712017 (555h16m38.262795052s) [upload=486813]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:45:38.691244197 (555h16m38.562327233s) [upload=486813]: Done uploading segments... Dec 28 18:45:38.691269095 (555h16m38.56235214s) [upload=486813]: Upload finished. err: <nil> Duration: 17.723313555s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.X........................X.....................X] duration: 1m0.00328303s trace: https://jaeger.dev.storj.io/trace/659b4940275e0f05 FAIL (2025-12-28 18:46:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== begin upload segment. Dec 28 18:45:25.180886421 (555h16m25.051969456s) [noise=true , address=145.79.227.21:4490 , piece_id=CLVXZXJRINCX6ZQP2CNAHLXRAOJI2WZUNAP4LSE2QZ5Y4OJ4AELA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.181484279 (555h16m25.052567314s) [noise=true , address=145.79.227.30:4768 , piece_id=GPXQOBUIPLRR3MAPDFDZUTL3D35BDNZIG573G4DAUGUH7DJU5T5A, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.181903237 (555h16m25.052986282s) [noise=true , address=77.90.31.15:4711 , piece_id=UPAPHGHWJBSBJYIDCN2FI3Z46T2JFUVAQOJPQPGTY4OHIWGG5VPA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182209721 (555h16m25.053292766s) [noise=true , address=64.49.13.27:4963 , piece_id=GRQVBULDQLYHV5ILRPMIWTMWP7DPPSJSKICIWEG4EKUXB7PZ7LSA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182531413 (555h16m25.053614448s) [noise=true , address=145.79.227.43:4160 , piece_id=X42ZZOWQLDUD34IWTT7R66VDHAJGA27IEUTAIMURDF4ELHLIW5PQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.182972543 (555h16m25.054055578s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183286231 (555h16m25.054369276s) [noise=true , address=45.91.116.8:4051 , piece_id=AY4G65PDYI34KUS4IAUYIK5WCU3VUMN7XCQJVZTXFXWDMGGYNPBQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183597022 (555h16m25.054680057s) [noise=true , address=77.90.31.21:4895 , piece_id=L5U5SMLBF4R3V7BUZLXRKQ2AEGIQGHPNEJL3FGGLF7DF5V2T4UIA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.183869430 (555h16m25.054952475s) [noise=true , address=145.79.227.40:4057 , piece_id=QRTNQJUDFQIOCWSZ7CD5OB5ZE4UJ7W2LKKITFJCS7GJ6G2EIAVWQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.185961945 (555h16m25.05704499s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.186737572 (555h16m25.057820617s) [noise=true , address=77.90.31.13:4630 , piece_id=63OQQKRAHKDKQ7PVN4UAIESK72NW6WOH3Q3QMAVZKSS32PATVKFQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.186816813 (555h16m25.057899848s) [noise=true , address=193.149.164.143:4696 , piece_id=4RTCBEIBUFN72SFD4VQ6UDCBEAU4DRF56HJIZT4L76ETKJUBWTZA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187247843 (555h16m25.058330878s) [noise=true , address=45.91.116.9:4281 , piece_id=AIALW5EEA2VRVELDNWXI64XNNUFQOGZ6424CBLXIOK4AD5S25ZSA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187406526 (555h16m25.058489561s) [noise=true , address=45.91.116.11:4130 , piece_id=ALKG6F653PTF6QYD3NTENX5VR2SE3HYTK4KQN6JF5AI2W5GII3CA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.187824422 (555h16m25.058907467s) [noise=true , address=145.79.227.44:4208 , piece_id=2SPAW6GFEDZWNDUCMWO6DNDYFVDGBJOL6XGQEFNTZBZNO7XPKXGQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188212400 (555h16m25.059295445s) [noise=true , address=45.91.116.4:4949 , piece_id=E745VQGK2CVZ2ETETNKIQWDODF4K6U7MDZKGW6SBHQZJEDIV2UXQ, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188670883 (555h16m25.059753928s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:25.188933434 (555h16m25.060016479s) [noise=true , address=77.90.31.18:4835 , piece_id=OQSXAPKIWHOMPWTP3A2LQ5B7DEI53CGPFLTLPZ4BDDOLZ3VVB5LA, seg_pos={0 0}, upload=486812]: Uploading piece... Dec 28 18:45:26.071888944 (555h16m25.942971989s) [noise=true , address=193.149.164.143:4696 , piece_id=4RTCBEIBUFN72SFD4VQ6UDCBEAU4DRF56HJIZT4L76ETKJUBWTZA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: storage node overloaded, request limit: 2000 Dec 28 18:45:26.652316958 (555h16m26.523400013s) [upload=486812]: Next returned nil segment Dec 28 18:45:26.652335714 (555h16m26.523418759s) [upload=486812]: Waiting for error group managing segments... Dec 28 18:45:27.070875407 (555h16m26.941958442s) [noise=true , address=145.79.227.21:4490 , piece_id=CLVXZXJRINCX6ZQP2CNAHLXRAOJI2WZUNAP4LSE2QZ5Y4OJ4AELA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.217018568 (555h16m27.088101603s) [noise=true , address=77.90.31.15:4711 , piece_id=UPAPHGHWJBSBJYIDCN2FI3Z46T2JFUVAQOJPQPGTY4OHIWGG5VPA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.244232238 (555h16m27.115315273s) [noise=true , address=77.90.31.21:4895 , piece_id=L5U5SMLBF4R3V7BUZLXRKQ2AEGIQGHPNEJL3FGGLF7DF5V2T4UIA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.264578651 (555h16m27.135661696s) [noise=true , address=77.90.31.13:4630 , piece_id=63OQQKRAHKDKQ7PVN4UAIESK72NW6WOH3Q3QMAVZKSS32PATVKFQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.277932088 (555h16m27.149015134s) [noise=true , address=45.91.116.9:4281 , piece_id=AIALW5EEA2VRVELDNWXI64XNNUFQOGZ6424CBLXIOK4AD5S25ZSA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.324815542 (555h16m27.195898577s) [noise=true , address=45.91.116.8:4051 , piece_id=AY4G65PDYI34KUS4IAUYIK5WCU3VUMN7XCQJVZTXFXWDMGGYNPBQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.329504170 (555h16m27.200587205s) [noise=true , address=77.90.31.18:4835 , piece_id=OQSXAPKIWHOMPWTP3A2LQ5B7DEI53CGPFLTLPZ4BDDOLZ3VVB5LA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.334083971 (555h16m27.205167006s) [noise=true , address=45.91.116.4:4949 , piece_id=E745VQGK2CVZ2ETETNKIQWDODF4K6U7MDZKGW6SBHQZJEDIV2UXQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.489154753 (555h16m27.360237798s) [noise=true , address=64.49.13.27:4963 , piece_id=GRQVBULDQLYHV5ILRPMIWTMWP7DPPSJSKICIWEG4EKUXB7PZ7LSA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.919436724 (555h16m27.790519759s) [noise=true , address=145.79.227.43:4160 , piece_id=X42ZZOWQLDUD34IWTT7R66VDHAJGA27IEUTAIMURDF4ELHLIW5PQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.987848773 (555h16m27.858931818s) [noise=true , address=145.79.227.30:4768 , piece_id=GPXQOBUIPLRR3MAPDFDZUTL3D35BDNZIG573G4DAUGUH7DJU5T5A, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:27.988333386 (555h16m27.859416421s) [noise=true , address=145.79.227.44:4208 , piece_id=2SPAW6GFEDZWNDUCMWO6DNDYFVDGBJOL6XGQEFNTZBZNO7XPKXGQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.044342256 (555h16m27.915425291s) [noise=true , address=145.79.227.40:4057 , piece_id=QRTNQJUDFQIOCWSZ7CD5OB5ZE4UJ7W2LKKITFJCS7GJ6G2EIAVWQ, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.051425193 (555h16m27.922508238s) [noise=true , address=45.91.116.11:4130 , piece_id=ALKG6F653PTF6QYD3NTENX5VR2SE3HYTK4KQN6JF5AI2W5GII3CA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: <nil> Dec 28 18:45:28.051449089 (555h16m27.922532144s) [seg_pos={0 0}, upload=486812]: Segment reached optimal threshold of 14 pieces. Dec 28 18:45:28.051675730 (555h16m27.922758775s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051686711 (555h16m27.922769756s) [noise=true , address=45.91.116.14:4199 , piece_id=6HKTDZ7GRMHE3KF6KQBX7YWC7S7BL7OULCQMLTGAKA6I7PEZHJ4Q, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051694917 (555h16m27.922777952s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051704274 (555h16m27.922787309s) [noise=true , address=145.79.227.34:4922 , piece_id=VYKXRVQAWG53AKQ7JCLHEW2ZLK4RTGCV6RA7H6Q7ZMGDHKWULURA, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051813111 (555h16m27.922896146s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1brZ4RtcK5J3xCPmnbsTccF9hrnj8tqhV1WyMqqrjuitgSGMou): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:28.051818422 (555h16m27.922901457s) [noise=true , address=145.79.227.41:4113 , piece_id=ZBBGKC5INYM64THZVVD3ONK7ZMPVLQQ2MXGKAXHHTZ637235EB3Q, seg_pos={0 0}, upload=486812]: Long tail cancellation due to optimal threshold Dec 28 18:45:28.051850843 (555h16m27.922933878s) [seg_pos={0 0}, upload=486812]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:45:28.051941646 (555h16m27.923024682s) [upload=486812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:45:28.051946335 (555h16m27.92302937s) [upload=486812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:45:28.356338310 (555h16m28.227421345s) [upload=486812]: Done uploading segments... Dec 28 18:45:28.356361995 (555h16m28.22744504s) [upload=486812]: Upload finished. err: <nil> Duration: 1m0.003302727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX.......X.X] duration: 30.004675571s trace: https://jaeger.dev.storj.io/trace/3390d4110367484a FAIL (2025-12-28 18:46:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: B434FCNTPVJWLJCNVBAH7PCVNV36VOH7D4LDPRNIJHL2S53VJ2MQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: MTO25OB5QF46F7RKL3VNCRN2FDUDZHXVEZR5WVU7UWRA3DSWCMQA) context deadline exceeded ======LOGS====== Dec 28 18:45:35.906443436 (555h16m35.777526471s) [upload=486813]: Starting upload Dec 28 18:45:35.906512087 (555h16m35.777595132s) [upload=486813]: Done starting upload Dec 28 18:45:35.906542686 (555h16m35.777625721s) [upload=486813]: Uploading segments... Dec 28 18:45:35.906746012 (555h16m35.777829057s) [upload=486813]: Got next segment. Inline: false Dec 28 18:45:35.906782863 (555h16m35.777865908s) [upload=486813]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:45:35.906787622 (555h16m35.777870667s) [upload=486813]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:45:36.160722747 (555h16m36.031805783s) [seg_pos={0 0}, upload=486813]: Begin upload segment... Dec 28 18:45:36.160766101 (555h16m36.031849146s) [seg_pos={0 0}, upload=486813]: Done begin upload segment. Dec 28 18:45:36.160808181 (555h16m36.031891246s) [noise=true , address=145.79.227.15:4873 , piece_id=3M22XHJMBI36BKXJQ7PAZC27OF6B25OUECQOQ3ENLWYXYUY53EHQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.160831936 (555h16m36.031914971s) [noise=true , address=145.79.227.50:4373 , piece_id=WAZLN6D7GJZGEH4X7HPSW464G2MWM6HFEI2BSWRIJPPUUKI5AJQQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.161276512 (555h16m36.032359547s) [noise=true , address=45.91.116.12:4225 , piece_id=GEUBFLBY3AQBZDQTCC2AFH5F7OAPCDUQ5VNLUBFV5TAXQGG5OT5Q, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.161647048 (555h16m36.032730083s) [noise=true , address=145.79.227.42:4121 , piece_id=R3RL7MUEZVKUWKSYUFIBWEIPYT3XX2EFW36FSNE7S3H4EF3WUXYQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.162121642 (555h16m36.033204677s) [noise=true , address=77.90.31.18:4808 , piece_id=B434FCNTPVJWLJCNVBAH7PCVNV36VOH7D4LDPRNIJHL2S53VJ2MQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.163634262 (555h16m36.034717297s) [noise=true , address=145.79.227.40:4051 , piece_id=COGK5Q54NMVRQGERW35NVXEKLR2MFFOSDA7MZ4BGROVC5PBR5OWA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.164506392 (555h16m36.035589438s) [noise=true , address=145.79.227.45:4231 , piece_id=WD7O7FZMM56OR3HNQXVPITQVDVNUKOTAUXQQSH7F2UYQLOMLSKMQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.164777329 (555h16m36.035860374s) [noise=true , address=145.79.227.26:4653 , piece_id=U64SHUAY77USOAV426LGCQEPFCJU7AA27BNFLZ3IZXQ6FQ4ZC3HQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.166056845 (555h16m36.0371399s) [noise=true , address=77.90.31.10:4519 , piece_id=GI4BU3XU4FEUGBIIOFX7XZQZANTZ6EOK4R3FZ2CEEBN4TK2VQPVQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.167280034 (555h16m36.038363069s) [noise=true , address=145.79.227.6:4259 , piece_id=7KFMHQYECMRO7V73XM46L4NWETQTBKAU7TMDGNKAXYY7CAGZISFA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.168392302 (555h16m36.039475347s) [noise=true , address=145.79.227.4:4186 , piece_id=5MVFCT46XABOVBSRGX3WQ2GBL6U6H5XDL6WNDGB3SDTVF3U2DMRA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.169570897 (555h16m36.040653932s) [noise=true , address=145.79.227.36:4517 , piece_id=YCCV4Y62VMPZYNLDNUMHLMCHYEOUBTLBRLE6HLLGZG43AFBIBOMA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.170281860 (555h16m36.041364905s) [noise=true , address=45.91.116.8:4047 , piece_id=MTO25OB5QF46F7RKL3VNCRN2FDUDZHXVEZR5WVU7UWRA3DSWCMQA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.171404928 (555h16m36.042487963s) [noise=true , address=45.91.116.10:4083 , piece_id=KAHNXBSMZZNMZCMDQ4WDIIRKDTUTAK6HHWOWCQBY34GHL2DUCOZQ, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.172599393 (555h16m36.043682429s) [noise=true , address=145.79.227.48:4303 , piece_id=22GPPEMVPSSUINZLSR34WF63LGYNTHK5ZV7T3S43CWXQKU7GD5RA, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:36.172818030 (555h16m36.043901075s) [noise=true , address=145.79.227.39:4013 , piece_id=6FLMCQDF5RUBHJAGEXP6N6CRBUFUIOQS5EQENGGCWERZ35XJNI6Q, seg_pos={0 0}, upload=486813]: Uploading piece... Dec 28 18:45:37.882334875 (555h16m37.75341792s) [upload=486813]: Next returned nil segment Dec 28 18:45:37.882380301 (555h16m37.753463366s) [upload=486813]: Waiting for error group managing segments... Dec 28 18:45:38.164456322 (555h16m38.035539357s) [noise=true , address=145.79.227.15:4873 , piece_id=3M22XHJMBI36BKXJQ7PAZC27OF6B25OUECQOQ3ENLWYXYUY53EHQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.166882873 (555h16m38.037965908s) [noise=true , address=145.79.227.4:4186 , piece_id=5MVFCT46XABOVBSRGX3WQ2GBL6U6H5XDL6WNDGB3SDTVF3U2DMRA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.169351003 (555h16m38.040434038s) [noise=true , address=145.79.227.6:4259 , piece_id=7KFMHQYECMRO7V73XM46L4NWETQTBKAU7TMDGNKAXYY7CAGZISFA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.170098216 (555h16m38.041181261s) [noise=true , address=145.79.227.36:4517 , piece_id=YCCV4Y62VMPZYNLDNUMHLMCHYEOUBTLBRLE6HLLGZG43AFBIBOMA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.175979766 (555h16m38.047062801s) [noise=true , address=145.79.227.39:4013 , piece_id=6FLMCQDF5RUBHJAGEXP6N6CRBUFUIOQS5EQENGGCWERZ35XJNI6Q, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.176308071 (555h16m38.047391106s) [noise=true , address=145.79.227.26:4653 , piece_id=U64SHUAY77USOAV426LGCQEPFCJU7AA27BNFLZ3IZXQ6FQ4ZC3HQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.183793214 (555h16m38.054876259s) [noise=true , address=145.79.227.42:4121 , piece_id=R3RL7MUEZVKUWKSYUFIBWEIPYT3XX2EFW36FSNE7S3H4EF3WUXYQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.200062653 (555h16m38.071145698s) [noise=true , address=145.79.227.48:4303 , piece_id=22GPPEMVPSSUINZLSR34WF63LGYNTHK5ZV7T3S43CWXQKU7GD5RA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.205211678 (555h16m38.076294723s) [noise=true , address=145.79.227.45:4231 , piece_id=WD7O7FZMM56OR3HNQXVPITQVDVNUKOTAUXQQSH7F2UYQLOMLSKMQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.207043045 (555h16m38.07812609s) [noise=true , address=145.79.227.50:4373 , piece_id=WAZLN6D7GJZGEH4X7HPSW464G2MWM6HFEI2BSWRIJPPUUKI5AJQQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.346274295 (555h16m38.21735734s) [noise=true , address=77.90.31.10:4519 , piece_id=GI4BU3XU4FEUGBIIOFX7XZQZANTZ6EOK4R3FZ2CEEBN4TK2VQPVQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.367812468 (555h16m38.238895503s) [noise=true , address=77.90.31.18:4808 , piece_id=B434FCNTPVJWLJCNVBAH7PCVNV36VOH7D4LDPRNIJHL2S53VJ2MQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.390916743 (555h16m38.261999778s) [noise=true , address=45.91.116.8:4047 , piece_id=MTO25OB5QF46F7RKL3VNCRN2FDUDZHXVEZR5WVU7UWRA3DSWCMQA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: <nil> Dec 28 18:45:38.390995512 (555h16m38.262078547s) [seg_pos={0 0}, upload=486813]: Segment reached optimal threshold of 13 pieces. Dec 28 18:45:38.391318237 (555h16m38.262401302s) [noise=true , address=45.91.116.10:4083 , piece_id=KAHNXBSMZZNMZCMDQ4WDIIRKDTUTAK6HHWOWCQBY34GHL2DUCOZQ, seg_pos={0 0}, upload=486813]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:38.391327164 (555h16m38.262410199s) [noise=true , address=45.91.116.10:4083 , piece_id=KAHNXBSMZZNMZCMDQ4WDIIRKDTUTAK6HHWOWCQBY34GHL2DUCOZQ, seg_pos={0 0}, upload=486813]: Long tail cancellation due to optimal threshold Dec 28 18:45:38.391538978 (555h16m38.262622023s) [noise=true , address=145.79.227.40:4051 , piece_id=COGK5Q54NMVRQGERW35NVXEKLR2MFFOSDA7MZ4BGROVC5PBR5OWA, seg_pos={0 0}, upload=486813]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:38.391561169 (555h16m38.262644224s) [noise=true , address=145.79.227.40:4051 , piece_id=COGK5Q54NMVRQGERW35NVXEKLR2MFFOSDA7MZ4BGROVC5PBR5OWA, seg_pos={0 0}, upload=486813]: Long tail cancellation due to optimal threshold Dec 28 18:45:38.391590916 (555h16m38.262673961s) [noise=true , address=45.91.116.12:4225 , piece_id=GEUBFLBY3AQBZDQTCC2AFH5F7OAPCDUQ5VNLUBFV5TAXQGG5OT5Q, seg_pos={0 0}, upload=486813]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 28 18:45:38.391600154 (555h16m38.262683199s) [noise=true , address=45.91.116.12:4225 , piece_id=GEUBFLBY3AQBZDQTCC2AFH5F7OAPCDUQ5VNLUBFV5TAXQGG5OT5Q, seg_pos={0 0}, upload=486813]: Long tail cancellation due to optimal threshold Dec 28 18:45:38.391625311 (555h16m38.262708357s) [seg_pos={0 0}, upload=486813]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:45:38.391707178 (555h16m38.262790213s) [upload=486813]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:45:38.391712017 (555h16m38.262795052s) [upload=486813]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:45:38.691244197 (555h16m38.562327233s) [upload=486813]: Done uploading segments... Dec 28 18:45:38.691269095 (555h16m38.56235214s) [upload=486813]: Upload finished. err: <nil> Duration: 30.004699889s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X.X........................X.....................] duration: 36.922508253s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:45:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 36.922539422s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX.......X.] duration: 5.721622599s trace: https://jaeger.dev.storj.io/trace/11a2d7ef2beffa97 FAIL (2025-12-28 18:44:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: X3Z46Q3PHHYMTMEEG4QJME4EFXMK5Y4VA3WZCQ52GWOEQTLRFBXA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: XF5FK4MWMSZH5YVKISVGHX3GZMJBVH2PFDBX6TAUXM3OZ42Y6Q3Q) context deadline exceeded ======LOGS====== gments... Dec 28 18:44:17.615675391 (555h15m17.486758426s) [upload=486797]: Got next segment. Inline: false Dec 28 18:44:17.615715056 (555h15m17.486798101s) [upload=486797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:44:17.615719525 (555h15m17.48680256s) [upload=486797]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:44:17.865577608 (555h15m17.736660653s) [seg_pos={0 0}, upload=486797]: Begin upload segment... Dec 28 18:44:17.865620239 (555h15m17.736703274s) [seg_pos={0 0}, upload=486797]: Done begin upload segment. Dec 28 18:44:17.865673861 (555h15m17.736756896s) [noise=true , address=145.79.227.19:4078 , piece_id=3MXANMGYOG7UK4V7OAURHGB74GEAZAPVYMHL5KNJXPEQJD57IYFA, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.865715020 (555h15m17.736798065s) [noise=true , address=45.91.116.10:4083 , piece_id=XF5FK4MWMSZH5YVKISVGHX3GZMJBVH2PFDBX6TAUXM3OZ42Y6Q3Q, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.866343016 (555h15m17.737426061s) [noise=true , address=45.91.116.6:4974 , piece_id=5G5XSA27L3TJUYKTMQIYU7KRES6JWQYR6A7ZT3PHER3YPWDUNZQA, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.866855391 (555h15m17.737938436s) [noise=true , address=145.79.227.39:4013 , piece_id=H4LHHC76D7RCYEBBAGUCXJHSJIHYPIPU3HLP5BPYJF57CNOYRUTQ, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.867282024 (555h15m17.738365069s) [noise=true , address=145.79.227.52:4445 , piece_id=Y2KNZO2XOF7I2D7KSUXZQ46MEGQCFMURWOS55BNVEPP24EU5ILKQ, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.867664983 (555h15m17.738748018s) [noise=true , address=145.79.227.37:4565 , piece_id=NCOT66AMXNTFAYK7WRGDCKSS2MPA54EBUD6XPQOBDSQWCXZC27PQ, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.868092837 (555h15m17.739175882s) [noise=true , address=145.79.227.9:4366 , piece_id=DDATECQOLLTIP7YZKRBWZXZ5IKIYXRVKZM76HW4ZNAGJPCCNHI6A, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.868458062 (555h15m17.739541107s) [noise=true , address=145.79.227.42:4123 , piece_id=XE6MMXRNNAGOJQ2IVGZPYYEACZCPTZ6SUBKCKBFIHJVLVOJZTPFQ, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.868866320 (555h15m17.739949355s) [noise=true , address=145.79.227.32:4835 , piece_id=WXZB77YNGRAAXCBGUPPYPCDCLSBPGJ2KX5NPPFME6MY4EM4LGEJA, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.869270971 (555h15m17.740354016s) [noise=true , address=145.79.227.45:4230 , piece_id=DCTLX2PDAKQJWMNTT4IH34VNV6FFIOIXXVDCKKDHTVWE6NMD5HCQ, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.870199388 (555h15m17.741282423s) [noise=true , address=145.79.227.21:4475 , piece_id=KTIXEFWNX3NPH3547VYU4CFUUGVYWXW3FRCD73Y2MGFO7L2BCNVA, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.870802527 (555h15m17.741885572s) [noise=true , address=45.91.116.7:4011 , piece_id=X3Z46Q3PHHYMTMEEG4QJME4EFXMK5Y4VA3WZCQ52GWOEQTLRFBXA, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.871206155 (555h15m17.7422892s) [noise=true , address=45.91.116.14:4191 , piece_id=3ZXNAZWJLV7KDMVRQ5PJRZSDQEFLVS5JX4V3GQOWO6HDT23AN3HQ, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.871689536 (555h15m17.742772571s) [noise=true , address=145.79.227.2:4113 , piece_id=I2ZPEBIVO7WYIQNE2LKMCPERL3ODMA4Z5XJUXQLRSEYJ2VNA2PEA, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.872028191 (555h15m17.743111226s) [noise=true , address=145.79.227.51:4410 , piece_id=K6XLDG2UEYE7MMX5KKNW6YQLMPU7DPFH7QTLIEK53NOHUQVW3VUA, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.872474461 (555h15m17.743557496s) [noise=true , address=145.79.227.7:4294 , piece_id=4K2XL7KFOQFIMU3EVAKXMNG2UNYS2SBX67OEXHE2N737D3C2EQXQ, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:19.486514002 (555h15m19.357597037s) [upload=486797]: Next returned nil segment Dec 28 18:44:19.486525263 (555h15m19.357608298s) [upload=486797]: Waiting for error group managing segments... Dec 28 18:44:19.871223695 (555h15m19.74230673s) [noise=true , address=145.79.227.9:4366 , piece_id=DDATECQOLLTIP7YZKRBWZXZ5IKIYXRVKZM76HW4ZNAGJPCCNHI6A, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.871861229 (555h15m19.742944305s) [noise=true , address=145.79.227.42:4123 , piece_id=XE6MMXRNNAGOJQ2IVGZPYYEACZCPTZ6SUBKCKBFIHJVLVOJZTPFQ, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.871919790 (555h15m19.743002835s) [noise=true , address=145.79.227.39:4013 , piece_id=H4LHHC76D7RCYEBBAGUCXJHSJIHYPIPU3HLP5BPYJF57CNOYRUTQ, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.872538168 (555h15m19.743621203s) [noise=true , address=145.79.227.51:4410 , piece_id=K6XLDG2UEYE7MMX5KKNW6YQLMPU7DPFH7QTLIEK53NOHUQVW3VUA, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.875086420 (555h15m19.746169455s) [noise=true , address=145.79.227.52:4445 , piece_id=Y2KNZO2XOF7I2D7KSUXZQ46MEGQCFMURWOS55BNVEPP24EU5ILKQ, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.876405081 (555h15m19.747488116s) [noise=true , address=145.79.227.19:4078 , piece_id=3MXANMGYOG7UK4V7OAURHGB74GEAZAPVYMHL5KNJXPEQJD57IYFA, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.889307069 (555h15m19.760390114s) [noise=true , address=145.79.227.45:4230 , piece_id=DCTLX2PDAKQJWMNTT4IH34VNV6FFIOIXXVDCKKDHTVWE6NMD5HCQ, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.897080962 (555h15m19.768163997s) [noise=true , address=145.79.227.21:4475 , piece_id=KTIXEFWNX3NPH3547VYU4CFUUGVYWXW3FRCD73Y2MGFO7L2BCNVA, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.900273722 (555h15m19.771356757s) [noise=true , address=145.79.227.37:4565 , piece_id=NCOT66AMXNTFAYK7WRGDCKSS2MPA54EBUD6XPQOBDSQWCXZC27PQ, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.901155540 (555h15m19.772238575s) [noise=true , address=145.79.227.32:4835 , piece_id=WXZB77YNGRAAXCBGUPPYPCDCLSBPGJ2KX5NPPFME6MY4EM4LGEJA, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:20.192433745 (555h15m20.06351678s) [noise=true , address=45.91.116.6:4974 , piece_id=5G5XSA27L3TJUYKTMQIYU7KRES6JWQYR6A7ZT3PHER3YPWDUNZQA, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:20.341545539 (555h15m20.212628584s) [noise=true , address=45.91.116.7:4011 , piece_id=X3Z46Q3PHHYMTMEEG4QJME4EFXMK5Y4VA3WZCQ52GWOEQTLRFBXA, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:20.567366170 (555h15m20.438449205s) [noise=true , address=45.91.116.10:4083 , piece_id=XF5FK4MWMSZH5YVKISVGHX3GZMJBVH2PFDBX6TAUXM3OZ42Y6Q3Q, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:20.567384716 (555h15m20.438467751s) [seg_pos={0 0}, upload=486797]: Segment reached optimal threshold of 13 pieces. Dec 28 18:44:20.567573365 (555h15m20.4386564s) [noise=true , address=145.79.227.7:4294 , piece_id=4K2XL7KFOQFIMU3EVAKXMNG2UNYS2SBX67OEXHE2N737D3C2EQXQ, seg_pos={0 0}, upload=486797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:54612->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:54612->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54612->145.79.227.7:4294: use of closed network connection Dec 28 18:44:20.567581891 (555h15m20.438664936s) [noise=true , address=145.79.227.7:4294 , piece_id=4K2XL7KFOQFIMU3EVAKXMNG2UNYS2SBX67OEXHE2N737D3C2EQXQ, seg_pos={0 0}, upload=486797]: Long tail cancellation due to optimal threshold Dec 28 18:44:20.567715576 (555h15m20.438798611s) [noise=true , address=145.79.227.2:4113 , piece_id=I2ZPEBIVO7WYIQNE2LKMCPERL3ODMA4Z5XJUXQLRSEYJ2VNA2PEA, seg_pos={0 0}, upload=486797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:51028->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:51028->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51028->145.79.227.2:4113: use of closed network connection Dec 28 18:44:20.567732598 (555h15m20.438815634s) [noise=true , address=145.79.227.2:4113 , piece_id=I2ZPEBIVO7WYIQNE2LKMCPERL3ODMA4Z5XJUXQLRSEYJ2VNA2PEA, seg_pos={0 0}, upload=486797]: Long tail cancellation due to optimal threshold Dec 28 18:44:20.567852035 (555h15m20.438935071s) [noise=true , address=45.91.116.14:4191 , piece_id=3ZXNAZWJLV7KDMVRQ5PJRZSDQEFLVS5JX4V3GQOWO6HDT23AN3HQ, seg_pos={0 0}, upload=486797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 28 18:44:20.567857386 (555h15m20.438940421s) [noise=true , address=45.91.116.14:4191 , piece_id=3ZXNAZWJLV7KDMVRQ5PJRZSDQEFLVS5JX4V3GQOWO6HDT23AN3HQ, seg_pos={0 0}, upload=486797]: Long tail cancellation due to optimal threshold Dec 28 18:44:20.567870852 (555h15m20.438953887s) [seg_pos={0 0}, upload=486797]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:44:20.567947127 (555h15m20.439030162s) [upload=486797]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:44:20.567951946 (555h15m20.439034981s) [upload=486797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:44:20.868410582 (555h15m20.739493627s) [upload=486797]: Done uploading segments... Dec 28 18:44:20.868432172 (555h15m20.739515207s) [upload=486797]: Upload finished. err: <nil> Duration: 5.721652867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX.......X] duration: 30.002740438s trace: https://jaeger.dev.storj.io/trace/11a2d7ef2beffa97 FAIL (2025-12-28 18:44:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: X3Z46Q3PHHYMTMEEG4QJME4EFXMK5Y4VA3WZCQ52GWOEQTLRFBXA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: XF5FK4MWMSZH5YVKISVGHX3GZMJBVH2PFDBX6TAUXM3OZ42Y6Q3Q) context deadline exceeded ======LOGS====== gments... Dec 28 18:44:17.615675391 (555h15m17.486758426s) [upload=486797]: Got next segment. Inline: false Dec 28 18:44:17.615715056 (555h15m17.486798101s) [upload=486797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:44:17.615719525 (555h15m17.48680256s) [upload=486797]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:44:17.865577608 (555h15m17.736660653s) [seg_pos={0 0}, upload=486797]: Begin upload segment... Dec 28 18:44:17.865620239 (555h15m17.736703274s) [seg_pos={0 0}, upload=486797]: Done begin upload segment. Dec 28 18:44:17.865673861 (555h15m17.736756896s) [noise=true , address=145.79.227.19:4078 , piece_id=3MXANMGYOG7UK4V7OAURHGB74GEAZAPVYMHL5KNJXPEQJD57IYFA, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.865715020 (555h15m17.736798065s) [noise=true , address=45.91.116.10:4083 , piece_id=XF5FK4MWMSZH5YVKISVGHX3GZMJBVH2PFDBX6TAUXM3OZ42Y6Q3Q, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.866343016 (555h15m17.737426061s) [noise=true , address=45.91.116.6:4974 , piece_id=5G5XSA27L3TJUYKTMQIYU7KRES6JWQYR6A7ZT3PHER3YPWDUNZQA, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.866855391 (555h15m17.737938436s) [noise=true , address=145.79.227.39:4013 , piece_id=H4LHHC76D7RCYEBBAGUCXJHSJIHYPIPU3HLP5BPYJF57CNOYRUTQ, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.867282024 (555h15m17.738365069s) [noise=true , address=145.79.227.52:4445 , piece_id=Y2KNZO2XOF7I2D7KSUXZQ46MEGQCFMURWOS55BNVEPP24EU5ILKQ, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.867664983 (555h15m17.738748018s) [noise=true , address=145.79.227.37:4565 , piece_id=NCOT66AMXNTFAYK7WRGDCKSS2MPA54EBUD6XPQOBDSQWCXZC27PQ, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.868092837 (555h15m17.739175882s) [noise=true , address=145.79.227.9:4366 , piece_id=DDATECQOLLTIP7YZKRBWZXZ5IKIYXRVKZM76HW4ZNAGJPCCNHI6A, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.868458062 (555h15m17.739541107s) [noise=true , address=145.79.227.42:4123 , piece_id=XE6MMXRNNAGOJQ2IVGZPYYEACZCPTZ6SUBKCKBFIHJVLVOJZTPFQ, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.868866320 (555h15m17.739949355s) [noise=true , address=145.79.227.32:4835 , piece_id=WXZB77YNGRAAXCBGUPPYPCDCLSBPGJ2KX5NPPFME6MY4EM4LGEJA, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.869270971 (555h15m17.740354016s) [noise=true , address=145.79.227.45:4230 , piece_id=DCTLX2PDAKQJWMNTT4IH34VNV6FFIOIXXVDCKKDHTVWE6NMD5HCQ, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.870199388 (555h15m17.741282423s) [noise=true , address=145.79.227.21:4475 , piece_id=KTIXEFWNX3NPH3547VYU4CFUUGVYWXW3FRCD73Y2MGFO7L2BCNVA, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.870802527 (555h15m17.741885572s) [noise=true , address=45.91.116.7:4011 , piece_id=X3Z46Q3PHHYMTMEEG4QJME4EFXMK5Y4VA3WZCQ52GWOEQTLRFBXA, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.871206155 (555h15m17.7422892s) [noise=true , address=45.91.116.14:4191 , piece_id=3ZXNAZWJLV7KDMVRQ5PJRZSDQEFLVS5JX4V3GQOWO6HDT23AN3HQ, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.871689536 (555h15m17.742772571s) [noise=true , address=145.79.227.2:4113 , piece_id=I2ZPEBIVO7WYIQNE2LKMCPERL3ODMA4Z5XJUXQLRSEYJ2VNA2PEA, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.872028191 (555h15m17.743111226s) [noise=true , address=145.79.227.51:4410 , piece_id=K6XLDG2UEYE7MMX5KKNW6YQLMPU7DPFH7QTLIEK53NOHUQVW3VUA, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:17.872474461 (555h15m17.743557496s) [noise=true , address=145.79.227.7:4294 , piece_id=4K2XL7KFOQFIMU3EVAKXMNG2UNYS2SBX67OEXHE2N737D3C2EQXQ, seg_pos={0 0}, upload=486797]: Uploading piece... Dec 28 18:44:19.486514002 (555h15m19.357597037s) [upload=486797]: Next returned nil segment Dec 28 18:44:19.486525263 (555h15m19.357608298s) [upload=486797]: Waiting for error group managing segments... Dec 28 18:44:19.871223695 (555h15m19.74230673s) [noise=true , address=145.79.227.9:4366 , piece_id=DDATECQOLLTIP7YZKRBWZXZ5IKIYXRVKZM76HW4ZNAGJPCCNHI6A, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.871861229 (555h15m19.742944305s) [noise=true , address=145.79.227.42:4123 , piece_id=XE6MMXRNNAGOJQ2IVGZPYYEACZCPTZ6SUBKCKBFIHJVLVOJZTPFQ, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.871919790 (555h15m19.743002835s) [noise=true , address=145.79.227.39:4013 , piece_id=H4LHHC76D7RCYEBBAGUCXJHSJIHYPIPU3HLP5BPYJF57CNOYRUTQ, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.872538168 (555h15m19.743621203s) [noise=true , address=145.79.227.51:4410 , piece_id=K6XLDG2UEYE7MMX5KKNW6YQLMPU7DPFH7QTLIEK53NOHUQVW3VUA, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.875086420 (555h15m19.746169455s) [noise=true , address=145.79.227.52:4445 , piece_id=Y2KNZO2XOF7I2D7KSUXZQ46MEGQCFMURWOS55BNVEPP24EU5ILKQ, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.876405081 (555h15m19.747488116s) [noise=true , address=145.79.227.19:4078 , piece_id=3MXANMGYOG7UK4V7OAURHGB74GEAZAPVYMHL5KNJXPEQJD57IYFA, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.889307069 (555h15m19.760390114s) [noise=true , address=145.79.227.45:4230 , piece_id=DCTLX2PDAKQJWMNTT4IH34VNV6FFIOIXXVDCKKDHTVWE6NMD5HCQ, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.897080962 (555h15m19.768163997s) [noise=true , address=145.79.227.21:4475 , piece_id=KTIXEFWNX3NPH3547VYU4CFUUGVYWXW3FRCD73Y2MGFO7L2BCNVA, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.900273722 (555h15m19.771356757s) [noise=true , address=145.79.227.37:4565 , piece_id=NCOT66AMXNTFAYK7WRGDCKSS2MPA54EBUD6XPQOBDSQWCXZC27PQ, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:19.901155540 (555h15m19.772238575s) [noise=true , address=145.79.227.32:4835 , piece_id=WXZB77YNGRAAXCBGUPPYPCDCLSBPGJ2KX5NPPFME6MY4EM4LGEJA, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:20.192433745 (555h15m20.06351678s) [noise=true , address=45.91.116.6:4974 , piece_id=5G5XSA27L3TJUYKTMQIYU7KRES6JWQYR6A7ZT3PHER3YPWDUNZQA, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:20.341545539 (555h15m20.212628584s) [noise=true , address=45.91.116.7:4011 , piece_id=X3Z46Q3PHHYMTMEEG4QJME4EFXMK5Y4VA3WZCQ52GWOEQTLRFBXA, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:20.567366170 (555h15m20.438449205s) [noise=true , address=45.91.116.10:4083 , piece_id=XF5FK4MWMSZH5YVKISVGHX3GZMJBVH2PFDBX6TAUXM3OZ42Y6Q3Q, seg_pos={0 0}, upload=486797]: Done uploading piece. err: <nil> Dec 28 18:44:20.567384716 (555h15m20.438467751s) [seg_pos={0 0}, upload=486797]: Segment reached optimal threshold of 13 pieces. Dec 28 18:44:20.567573365 (555h15m20.4386564s) [noise=true , address=145.79.227.7:4294 , piece_id=4K2XL7KFOQFIMU3EVAKXMNG2UNYS2SBX67OEXHE2N737D3C2EQXQ, seg_pos={0 0}, upload=486797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:54612->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:54612->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54612->145.79.227.7:4294: use of closed network connection Dec 28 18:44:20.567581891 (555h15m20.438664936s) [noise=true , address=145.79.227.7:4294 , piece_id=4K2XL7KFOQFIMU3EVAKXMNG2UNYS2SBX67OEXHE2N737D3C2EQXQ, seg_pos={0 0}, upload=486797]: Long tail cancellation due to optimal threshold Dec 28 18:44:20.567715576 (555h15m20.438798611s) [noise=true , address=145.79.227.2:4113 , piece_id=I2ZPEBIVO7WYIQNE2LKMCPERL3ODMA4Z5XJUXQLRSEYJ2VNA2PEA, seg_pos={0 0}, upload=486797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:51028->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:51028->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51028->145.79.227.2:4113: use of closed network connection Dec 28 18:44:20.567732598 (555h15m20.438815634s) [noise=true , address=145.79.227.2:4113 , piece_id=I2ZPEBIVO7WYIQNE2LKMCPERL3ODMA4Z5XJUXQLRSEYJ2VNA2PEA, seg_pos={0 0}, upload=486797]: Long tail cancellation due to optimal threshold Dec 28 18:44:20.567852035 (555h15m20.438935071s) [noise=true , address=45.91.116.14:4191 , piece_id=3ZXNAZWJLV7KDMVRQ5PJRZSDQEFLVS5JX4V3GQOWO6HDT23AN3HQ, seg_pos={0 0}, upload=486797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 28 18:44:20.567857386 (555h15m20.438940421s) [noise=true , address=45.91.116.14:4191 , piece_id=3ZXNAZWJLV7KDMVRQ5PJRZSDQEFLVS5JX4V3GQOWO6HDT23AN3HQ, seg_pos={0 0}, upload=486797]: Long tail cancellation due to optimal threshold Dec 28 18:44:20.567870852 (555h15m20.438953887s) [seg_pos={0 0}, upload=486797]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:44:20.567947127 (555h15m20.439030162s) [upload=486797]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:44:20.567951946 (555h15m20.439034981s) [upload=486797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:44:20.868410582 (555h15m20.739493627s) [upload=486797]: Done uploading segments... Dec 28 18:44:20.868432172 (555h15m20.739515207s) [upload=486797]: Upload finished. err: <nil> Duration: 30.002761077s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X.X........................X....................] duration: 29.678660974s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:44:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 29.678693927s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX.......] duration: 8.057899033s trace: https://jaeger.dev.storj.io/trace/785dcaf51e874964 FAIL (2025-12-28 18:43:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA) context deadline exceeded ======LOGS====== 9m49.795105852s) [upload=486719]: Uploading segments... Dec 28 18:38:49.924182661 (555h9m49.795265696s) [upload=486719]: Got next segment. Inline: false Dec 28 18:38:49.924242615 (555h9m49.79532566s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:38:49.924247405 (555h9m49.79533045s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:38:50.170791617 (555h9m50.041874682s) [seg_pos={0 0}, upload=486719]: Begin upload segment... Dec 28 18:38:50.170857693 (555h9m50.041940748s) [seg_pos={0 0}, upload=486719]: Done begin upload segment. Dec 28 18:38:50.170919190 (555h9m50.042002225s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.170984374 (555h9m50.04206742s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171606730 (555h9m50.042689765s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171837629 (555h9m50.042920664s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172032620 (555h9m50.043115666s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172435748 (555h9m50.043518784s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172785054 (555h9m50.043868099s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173070988 (555h9m50.044154033s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173379245 (555h9m50.04446228s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173593743 (555h9m50.044676778s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173951284 (555h9m50.045034319s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174139122 (555h9m50.045222157s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174493948 (555h9m50.045576983s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174752059 (555h9m50.045835094s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175082959 (555h9m50.046165994s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175143824 (555h9m50.046226859s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:51.887343999 (555h9m51.758427034s) [upload=486719]: Next returned nil segment Dec 28 18:38:51.887358918 (555h9m51.758441963s) [upload=486719]: Waiting for error group managing segments... Dec 28 18:38:52.172766574 (555h9m52.043849609s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.176871520 (555h9m52.047954565s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.182783407 (555h9m52.053866452s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.189914355 (555h9m52.06099739s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.193182439 (555h9m52.064265474s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.194753169 (555h9m52.065836204s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.195031610 (555h9m52.066114645s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.198391939 (555h9m52.069474974s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.209163559 (555h9m52.080246604s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.402848580 (555h9m52.273931625s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.449323697 (555h9m52.320406742s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.520405688 (555h9m52.391488733s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655345046 (555h9m52.526428082s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655373009 (555h9m52.526456044s) [seg_pos={0 0}, upload=486719]: Segment reached optimal threshold of 13 pieces. Dec 28 18:38:52.655557942 (555h9m52.526640977s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection Dec 28 18:38:52.655563672 (555h9m52.526646708s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655657632 (555h9m52.526740667s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection Dec 28 18:38:52.655664174 (555h9m52.526747209s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655775666 (555h9m52.526858972s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 18:38:52.655798770 (555h9m52.526881805s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655860497 (555h9m52.526943542s) [seg_pos={0 0}, upload=486719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:38:52.656024640 (555h9m52.527107675s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:38:52.656042815 (555h9m52.52712587s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:38:52.951505516 (555h9m52.822588561s) [upload=486719]: Done uploading segments... Dec 28 18:38:52.951531356 (555h9m52.822614401s) [upload=486719]: Upload finished. err: <nil> Duration: 8.057927998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.X........................X...................] duration: 15.828875405s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:43:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 15.828905132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 3.4100634s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:43:08 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.410096283s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX......] duration: 5.670931805s trace: https://jaeger.dev.storj.io/trace/785dcaf51e874964 FAIL (2025-12-28 18:43:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA) context deadline exceeded ======LOGS====== 9m49.795105852s) [upload=486719]: Uploading segments... Dec 28 18:38:49.924182661 (555h9m49.795265696s) [upload=486719]: Got next segment. Inline: false Dec 28 18:38:49.924242615 (555h9m49.79532566s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:38:49.924247405 (555h9m49.79533045s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:38:50.170791617 (555h9m50.041874682s) [seg_pos={0 0}, upload=486719]: Begin upload segment... Dec 28 18:38:50.170857693 (555h9m50.041940748s) [seg_pos={0 0}, upload=486719]: Done begin upload segment. Dec 28 18:38:50.170919190 (555h9m50.042002225s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.170984374 (555h9m50.04206742s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171606730 (555h9m50.042689765s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171837629 (555h9m50.042920664s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172032620 (555h9m50.043115666s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172435748 (555h9m50.043518784s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172785054 (555h9m50.043868099s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173070988 (555h9m50.044154033s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173379245 (555h9m50.04446228s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173593743 (555h9m50.044676778s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173951284 (555h9m50.045034319s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174139122 (555h9m50.045222157s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174493948 (555h9m50.045576983s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174752059 (555h9m50.045835094s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175082959 (555h9m50.046165994s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175143824 (555h9m50.046226859s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:51.887343999 (555h9m51.758427034s) [upload=486719]: Next returned nil segment Dec 28 18:38:51.887358918 (555h9m51.758441963s) [upload=486719]: Waiting for error group managing segments... Dec 28 18:38:52.172766574 (555h9m52.043849609s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.176871520 (555h9m52.047954565s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.182783407 (555h9m52.053866452s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.189914355 (555h9m52.06099739s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.193182439 (555h9m52.064265474s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.194753169 (555h9m52.065836204s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.195031610 (555h9m52.066114645s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.198391939 (555h9m52.069474974s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.209163559 (555h9m52.080246604s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.402848580 (555h9m52.273931625s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.449323697 (555h9m52.320406742s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.520405688 (555h9m52.391488733s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655345046 (555h9m52.526428082s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655373009 (555h9m52.526456044s) [seg_pos={0 0}, upload=486719]: Segment reached optimal threshold of 13 pieces. Dec 28 18:38:52.655557942 (555h9m52.526640977s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection Dec 28 18:38:52.655563672 (555h9m52.526646708s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655657632 (555h9m52.526740667s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection Dec 28 18:38:52.655664174 (555h9m52.526747209s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655775666 (555h9m52.526858972s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 18:38:52.655798770 (555h9m52.526881805s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655860497 (555h9m52.526943542s) [seg_pos={0 0}, upload=486719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:38:52.656024640 (555h9m52.527107675s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:38:52.656042815 (555h9m52.52712587s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:38:52.951505516 (555h9m52.822588561s) [upload=486719]: Done uploading segments... Dec 28 18:38:52.951531356 (555h9m52.822614401s) [upload=486719]: Upload finished. err: <nil> Duration: 5.670959838s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.............X.X........................X..................] duration: 34.295589145s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:42:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 34.295620563s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 3.522067125s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:42:27 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.522094567s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX.....] duration: 6.397087453s trace: https://jaeger.dev.storj.io/trace/785dcaf51e874964 FAIL (2025-12-28 18:42:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA) context deadline exceeded ======LOGS====== 9m49.795105852s) [upload=486719]: Uploading segments... Dec 28 18:38:49.924182661 (555h9m49.795265696s) [upload=486719]: Got next segment. Inline: false Dec 28 18:38:49.924242615 (555h9m49.79532566s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:38:49.924247405 (555h9m49.79533045s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:38:50.170791617 (555h9m50.041874682s) [seg_pos={0 0}, upload=486719]: Begin upload segment... Dec 28 18:38:50.170857693 (555h9m50.041940748s) [seg_pos={0 0}, upload=486719]: Done begin upload segment. Dec 28 18:38:50.170919190 (555h9m50.042002225s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.170984374 (555h9m50.04206742s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171606730 (555h9m50.042689765s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171837629 (555h9m50.042920664s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172032620 (555h9m50.043115666s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172435748 (555h9m50.043518784s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172785054 (555h9m50.043868099s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173070988 (555h9m50.044154033s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173379245 (555h9m50.04446228s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173593743 (555h9m50.044676778s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173951284 (555h9m50.045034319s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174139122 (555h9m50.045222157s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174493948 (555h9m50.045576983s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174752059 (555h9m50.045835094s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175082959 (555h9m50.046165994s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175143824 (555h9m50.046226859s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:51.887343999 (555h9m51.758427034s) [upload=486719]: Next returned nil segment Dec 28 18:38:51.887358918 (555h9m51.758441963s) [upload=486719]: Waiting for error group managing segments... Dec 28 18:38:52.172766574 (555h9m52.043849609s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.176871520 (555h9m52.047954565s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.182783407 (555h9m52.053866452s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.189914355 (555h9m52.06099739s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.193182439 (555h9m52.064265474s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.194753169 (555h9m52.065836204s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.195031610 (555h9m52.066114645s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.198391939 (555h9m52.069474974s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.209163559 (555h9m52.080246604s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.402848580 (555h9m52.273931625s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.449323697 (555h9m52.320406742s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.520405688 (555h9m52.391488733s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655345046 (555h9m52.526428082s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655373009 (555h9m52.526456044s) [seg_pos={0 0}, upload=486719]: Segment reached optimal threshold of 13 pieces. Dec 28 18:38:52.655557942 (555h9m52.526640977s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection Dec 28 18:38:52.655563672 (555h9m52.526646708s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655657632 (555h9m52.526740667s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection Dec 28 18:38:52.655664174 (555h9m52.526747209s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655775666 (555h9m52.526858972s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 18:38:52.655798770 (555h9m52.526881805s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655860497 (555h9m52.526943542s) [seg_pos={0 0}, upload=486719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:38:52.656024640 (555h9m52.527107675s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:38:52.656042815 (555h9m52.52712587s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:38:52.951505516 (555h9m52.822588561s) [upload=486719]: Done uploading segments... Dec 28 18:38:52.951531356 (555h9m52.822614401s) [upload=486719]: Upload finished. err: <nil> Duration: 6.397113192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.............X.X........................X.................] duration: 35.31266352s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:41:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 35.312695762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 3.257122605s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:41:20 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.257152514s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX....] duration: 7.817087565s trace: https://jaeger.dev.storj.io/trace/785dcaf51e874964 FAIL (2025-12-28 18:41:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA) context deadline exceeded ======LOGS====== 9m49.795105852s) [upload=486719]: Uploading segments... Dec 28 18:38:49.924182661 (555h9m49.795265696s) [upload=486719]: Got next segment. Inline: false Dec 28 18:38:49.924242615 (555h9m49.79532566s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:38:49.924247405 (555h9m49.79533045s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:38:50.170791617 (555h9m50.041874682s) [seg_pos={0 0}, upload=486719]: Begin upload segment... Dec 28 18:38:50.170857693 (555h9m50.041940748s) [seg_pos={0 0}, upload=486719]: Done begin upload segment. Dec 28 18:38:50.170919190 (555h9m50.042002225s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.170984374 (555h9m50.04206742s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171606730 (555h9m50.042689765s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171837629 (555h9m50.042920664s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172032620 (555h9m50.043115666s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172435748 (555h9m50.043518784s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172785054 (555h9m50.043868099s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173070988 (555h9m50.044154033s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173379245 (555h9m50.04446228s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173593743 (555h9m50.044676778s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173951284 (555h9m50.045034319s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174139122 (555h9m50.045222157s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174493948 (555h9m50.045576983s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174752059 (555h9m50.045835094s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175082959 (555h9m50.046165994s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175143824 (555h9m50.046226859s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:51.887343999 (555h9m51.758427034s) [upload=486719]: Next returned nil segment Dec 28 18:38:51.887358918 (555h9m51.758441963s) [upload=486719]: Waiting for error group managing segments... Dec 28 18:38:52.172766574 (555h9m52.043849609s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.176871520 (555h9m52.047954565s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.182783407 (555h9m52.053866452s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.189914355 (555h9m52.06099739s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.193182439 (555h9m52.064265474s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.194753169 (555h9m52.065836204s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.195031610 (555h9m52.066114645s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.198391939 (555h9m52.069474974s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.209163559 (555h9m52.080246604s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.402848580 (555h9m52.273931625s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.449323697 (555h9m52.320406742s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.520405688 (555h9m52.391488733s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655345046 (555h9m52.526428082s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655373009 (555h9m52.526456044s) [seg_pos={0 0}, upload=486719]: Segment reached optimal threshold of 13 pieces. Dec 28 18:38:52.655557942 (555h9m52.526640977s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection Dec 28 18:38:52.655563672 (555h9m52.526646708s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655657632 (555h9m52.526740667s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection Dec 28 18:38:52.655664174 (555h9m52.526747209s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655775666 (555h9m52.526858972s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 18:38:52.655798770 (555h9m52.526881805s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655860497 (555h9m52.526943542s) [seg_pos={0 0}, upload=486719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:38:52.656024640 (555h9m52.527107675s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:38:52.656042815 (555h9m52.52712587s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:38:52.951505516 (555h9m52.822588561s) [upload=486719]: Done uploading segments... Dec 28 18:38:52.951531356 (555h9m52.822614401s) [upload=486719]: Upload finished. err: <nil> Duration: 7.817120047s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX...] duration: 5.874268183s trace: https://jaeger.dev.storj.io/trace/785dcaf51e874964 FAIL (2025-12-28 18:40:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA) context deadline exceeded ======LOGS====== 9m49.795105852s) [upload=486719]: Uploading segments... Dec 28 18:38:49.924182661 (555h9m49.795265696s) [upload=486719]: Got next segment. Inline: false Dec 28 18:38:49.924242615 (555h9m49.79532566s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:38:49.924247405 (555h9m49.79533045s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:38:50.170791617 (555h9m50.041874682s) [seg_pos={0 0}, upload=486719]: Begin upload segment... Dec 28 18:38:50.170857693 (555h9m50.041940748s) [seg_pos={0 0}, upload=486719]: Done begin upload segment. Dec 28 18:38:50.170919190 (555h9m50.042002225s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.170984374 (555h9m50.04206742s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171606730 (555h9m50.042689765s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171837629 (555h9m50.042920664s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172032620 (555h9m50.043115666s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172435748 (555h9m50.043518784s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172785054 (555h9m50.043868099s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173070988 (555h9m50.044154033s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173379245 (555h9m50.04446228s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173593743 (555h9m50.044676778s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173951284 (555h9m50.045034319s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174139122 (555h9m50.045222157s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174493948 (555h9m50.045576983s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174752059 (555h9m50.045835094s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175082959 (555h9m50.046165994s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175143824 (555h9m50.046226859s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:51.887343999 (555h9m51.758427034s) [upload=486719]: Next returned nil segment Dec 28 18:38:51.887358918 (555h9m51.758441963s) [upload=486719]: Waiting for error group managing segments... Dec 28 18:38:52.172766574 (555h9m52.043849609s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.176871520 (555h9m52.047954565s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.182783407 (555h9m52.053866452s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.189914355 (555h9m52.06099739s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.193182439 (555h9m52.064265474s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.194753169 (555h9m52.065836204s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.195031610 (555h9m52.066114645s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.198391939 (555h9m52.069474974s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.209163559 (555h9m52.080246604s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.402848580 (555h9m52.273931625s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.449323697 (555h9m52.320406742s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.520405688 (555h9m52.391488733s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655345046 (555h9m52.526428082s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655373009 (555h9m52.526456044s) [seg_pos={0 0}, upload=486719]: Segment reached optimal threshold of 13 pieces. Dec 28 18:38:52.655557942 (555h9m52.526640977s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection Dec 28 18:38:52.655563672 (555h9m52.526646708s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655657632 (555h9m52.526740667s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection Dec 28 18:38:52.655664174 (555h9m52.526747209s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655775666 (555h9m52.526858972s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 18:38:52.655798770 (555h9m52.526881805s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655860497 (555h9m52.526943542s) [seg_pos={0 0}, upload=486719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:38:52.656024640 (555h9m52.527107675s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:38:52.656042815 (555h9m52.52712587s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:38:52.951505516 (555h9m52.822588561s) [upload=486719]: Done uploading segments... Dec 28 18:38:52.951531356 (555h9m52.822614401s) [upload=486719]: Upload finished. err: <nil> Duration: 5.874293421s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 3.486170943s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:40:58 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.486200048s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.............X.X........................X................] duration: 46.200190898s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:40:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 46.200210846s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX..] duration: 19.778656854s trace: https://jaeger.dev.storj.io/trace/785dcaf51e874964 FAIL (2025-12-28 18:40:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA) context deadline exceeded ======LOGS====== 9m49.795105852s) [upload=486719]: Uploading segments... Dec 28 18:38:49.924182661 (555h9m49.795265696s) [upload=486719]: Got next segment. Inline: false Dec 28 18:38:49.924242615 (555h9m49.79532566s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:38:49.924247405 (555h9m49.79533045s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:38:50.170791617 (555h9m50.041874682s) [seg_pos={0 0}, upload=486719]: Begin upload segment... Dec 28 18:38:50.170857693 (555h9m50.041940748s) [seg_pos={0 0}, upload=486719]: Done begin upload segment. Dec 28 18:38:50.170919190 (555h9m50.042002225s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.170984374 (555h9m50.04206742s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171606730 (555h9m50.042689765s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171837629 (555h9m50.042920664s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172032620 (555h9m50.043115666s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172435748 (555h9m50.043518784s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172785054 (555h9m50.043868099s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173070988 (555h9m50.044154033s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173379245 (555h9m50.04446228s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173593743 (555h9m50.044676778s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173951284 (555h9m50.045034319s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174139122 (555h9m50.045222157s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174493948 (555h9m50.045576983s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174752059 (555h9m50.045835094s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175082959 (555h9m50.046165994s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175143824 (555h9m50.046226859s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:51.887343999 (555h9m51.758427034s) [upload=486719]: Next returned nil segment Dec 28 18:38:51.887358918 (555h9m51.758441963s) [upload=486719]: Waiting for error group managing segments... Dec 28 18:38:52.172766574 (555h9m52.043849609s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.176871520 (555h9m52.047954565s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.182783407 (555h9m52.053866452s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.189914355 (555h9m52.06099739s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.193182439 (555h9m52.064265474s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.194753169 (555h9m52.065836204s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.195031610 (555h9m52.066114645s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.198391939 (555h9m52.069474974s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.209163559 (555h9m52.080246604s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.402848580 (555h9m52.273931625s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.449323697 (555h9m52.320406742s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.520405688 (555h9m52.391488733s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655345046 (555h9m52.526428082s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655373009 (555h9m52.526456044s) [seg_pos={0 0}, upload=486719]: Segment reached optimal threshold of 13 pieces. Dec 28 18:38:52.655557942 (555h9m52.526640977s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection Dec 28 18:38:52.655563672 (555h9m52.526646708s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655657632 (555h9m52.526740667s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection Dec 28 18:38:52.655664174 (555h9m52.526747209s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655775666 (555h9m52.526858972s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 18:38:52.655798770 (555h9m52.526881805s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655860497 (555h9m52.526943542s) [seg_pos={0 0}, upload=486719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:38:52.656024640 (555h9m52.527107675s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:38:52.656042815 (555h9m52.52712587s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:38:52.951505516 (555h9m52.822588561s) [upload=486719]: Done uploading segments... Dec 28 18:38:52.951531356 (555h9m52.822614401s) [upload=486719]: Upload finished. err: <nil> Duration: 19.778695909s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 3.42082354s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:40:07 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.420861232s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.............X.X........................X...............] duration: 27.567127762s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:40:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 27.567274682s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX.] duration: 6.474394087s trace: https://jaeger.dev.storj.io/trace/785dcaf51e874964 FAIL (2025-12-28 18:39:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA) context deadline exceeded ======LOGS====== 9m49.795105852s) [upload=486719]: Uploading segments... Dec 28 18:38:49.924182661 (555h9m49.795265696s) [upload=486719]: Got next segment. Inline: false Dec 28 18:38:49.924242615 (555h9m49.79532566s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:38:49.924247405 (555h9m49.79533045s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:38:50.170791617 (555h9m50.041874682s) [seg_pos={0 0}, upload=486719]: Begin upload segment... Dec 28 18:38:50.170857693 (555h9m50.041940748s) [seg_pos={0 0}, upload=486719]: Done begin upload segment. Dec 28 18:38:50.170919190 (555h9m50.042002225s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.170984374 (555h9m50.04206742s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171606730 (555h9m50.042689765s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171837629 (555h9m50.042920664s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172032620 (555h9m50.043115666s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172435748 (555h9m50.043518784s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172785054 (555h9m50.043868099s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173070988 (555h9m50.044154033s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173379245 (555h9m50.04446228s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173593743 (555h9m50.044676778s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173951284 (555h9m50.045034319s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174139122 (555h9m50.045222157s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174493948 (555h9m50.045576983s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174752059 (555h9m50.045835094s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175082959 (555h9m50.046165994s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175143824 (555h9m50.046226859s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:51.887343999 (555h9m51.758427034s) [upload=486719]: Next returned nil segment Dec 28 18:38:51.887358918 (555h9m51.758441963s) [upload=486719]: Waiting for error group managing segments... Dec 28 18:38:52.172766574 (555h9m52.043849609s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.176871520 (555h9m52.047954565s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.182783407 (555h9m52.053866452s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.189914355 (555h9m52.06099739s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.193182439 (555h9m52.064265474s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.194753169 (555h9m52.065836204s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.195031610 (555h9m52.066114645s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.198391939 (555h9m52.069474974s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.209163559 (555h9m52.080246604s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.402848580 (555h9m52.273931625s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.449323697 (555h9m52.320406742s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.520405688 (555h9m52.391488733s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655345046 (555h9m52.526428082s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655373009 (555h9m52.526456044s) [seg_pos={0 0}, upload=486719]: Segment reached optimal threshold of 13 pieces. Dec 28 18:38:52.655557942 (555h9m52.526640977s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection Dec 28 18:38:52.655563672 (555h9m52.526646708s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655657632 (555h9m52.526740667s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection Dec 28 18:38:52.655664174 (555h9m52.526747209s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655775666 (555h9m52.526858972s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 18:38:52.655798770 (555h9m52.526881805s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655860497 (555h9m52.526943542s) [seg_pos={0 0}, upload=486719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:38:52.656024640 (555h9m52.527107675s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:38:52.656042815 (555h9m52.52712587s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:38:52.951505516 (555h9m52.822588561s) [upload=486719]: Done uploading segments... Dec 28 18:38:52.951531356 (555h9m52.822614401s) [upload=486719]: Upload finished. err: <nil> Duration: 6.47442231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.............X.X........................X..............] duration: 18.829213187s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:39:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 18.829239257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX] duration: 30.002671516s trace: https://jaeger.dev.storj.io/trace/785dcaf51e874964 FAIL (2025-12-28 18:39:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA) context deadline exceeded ======LOGS====== 9m49.795105852s) [upload=486719]: Uploading segments... Dec 28 18:38:49.924182661 (555h9m49.795265696s) [upload=486719]: Got next segment. Inline: false Dec 28 18:38:49.924242615 (555h9m49.79532566s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:38:49.924247405 (555h9m49.79533045s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:38:50.170791617 (555h9m50.041874682s) [seg_pos={0 0}, upload=486719]: Begin upload segment... Dec 28 18:38:50.170857693 (555h9m50.041940748s) [seg_pos={0 0}, upload=486719]: Done begin upload segment. Dec 28 18:38:50.170919190 (555h9m50.042002225s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.170984374 (555h9m50.04206742s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171606730 (555h9m50.042689765s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171837629 (555h9m50.042920664s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172032620 (555h9m50.043115666s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172435748 (555h9m50.043518784s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172785054 (555h9m50.043868099s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173070988 (555h9m50.044154033s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173379245 (555h9m50.04446228s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173593743 (555h9m50.044676778s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173951284 (555h9m50.045034319s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174139122 (555h9m50.045222157s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174493948 (555h9m50.045576983s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174752059 (555h9m50.045835094s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175082959 (555h9m50.046165994s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175143824 (555h9m50.046226859s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:51.887343999 (555h9m51.758427034s) [upload=486719]: Next returned nil segment Dec 28 18:38:51.887358918 (555h9m51.758441963s) [upload=486719]: Waiting for error group managing segments... Dec 28 18:38:52.172766574 (555h9m52.043849609s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.176871520 (555h9m52.047954565s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.182783407 (555h9m52.053866452s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.189914355 (555h9m52.06099739s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.193182439 (555h9m52.064265474s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.194753169 (555h9m52.065836204s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.195031610 (555h9m52.066114645s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.198391939 (555h9m52.069474974s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.209163559 (555h9m52.080246604s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.402848580 (555h9m52.273931625s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.449323697 (555h9m52.320406742s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.520405688 (555h9m52.391488733s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655345046 (555h9m52.526428082s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655373009 (555h9m52.526456044s) [seg_pos={0 0}, upload=486719]: Segment reached optimal threshold of 13 pieces. Dec 28 18:38:52.655557942 (555h9m52.526640977s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection Dec 28 18:38:52.655563672 (555h9m52.526646708s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655657632 (555h9m52.526740667s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection Dec 28 18:38:52.655664174 (555h9m52.526747209s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655775666 (555h9m52.526858972s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 18:38:52.655798770 (555h9m52.526881805s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655860497 (555h9m52.526943542s) [seg_pos={0 0}, upload=486719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:38:52.656024640 (555h9m52.527107675s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:38:52.656042815 (555h9m52.52712587s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:38:52.951505516 (555h9m52.822588561s) [upload=486719]: Done uploading segments... Dec 28 18:38:52.951531356 (555h9m52.822614401s) [upload=486719]: Upload finished. err: <nil> Duration: 30.002693608s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 3.390520938s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:39:04 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.390551416s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 3.516215072s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:38:40 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.516241894s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.............X.X........................X.............] duration: 33.628723708s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:38:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 33.62873975s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...X] duration: 30.002813589s trace: https://jaeger.dev.storj.io/trace/4fbed146ec31b60f FAIL (2025-12-28 18:38:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: O45R525ETFGWT43PDWDUJV5VYPIKXUDHDHR4JKAKPOWZ6BRUGULA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: UIPUC3NJRS4S6BAASNOF7E4LUAMJFCJUT76FRPAMWJH6PXD344VA) context deadline exceeded ======LOGS====== Dec 28 18:37:47.952924132 (555h8m47.824007167s) [upload=486703]: Starting upload Dec 28 18:37:47.952996239 (555h8m47.824079274s) [upload=486703]: Done starting upload Dec 28 18:37:47.953044731 (555h8m47.824127766s) [upload=486703]: Uploading segments... Dec 28 18:37:47.953210126 (555h8m47.824293171s) [upload=486703]: Got next segment. Inline: false Dec 28 18:37:47.953242157 (555h8m47.824325192s) [upload=486703]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:37:47.953247898 (555h8m47.824330933s) [upload=486703]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:37:48.203667200 (555h8m48.074750265s) [seg_pos={0 0}, upload=486703]: Begin upload segment... Dec 28 18:37:48.203720922 (555h8m48.074803957s) [seg_pos={0 0}, upload=486703]: Done begin upload segment. Dec 28 18:37:48.203734067 (555h8m48.074817112s) [noise=true , address=77.90.31.11:4556 , piece_id=JZTPKCAMECPIJKWWXJCANZRXUCMIDRELAUXZTF537ETNSBOJMRMQ, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.203749928 (555h8m48.074832963s) [noise=true , address=45.91.116.5:4298 , piece_id=O45R525ETFGWT43PDWDUJV5VYPIKXUDHDHR4JKAKPOWZ6BRUGULA, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.203916956 (555h8m48.074999991s) [noise=true , address=145.79.227.36:4519 , piece_id=LHQKDQ6XHMOZFOU66MMMLKBF2W5R6WY7UQ6WLAUKPQOM4KNYFGGQ, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.204281420 (555h8m48.075364455s) [noise=true , address=145.79.227.16:4911 , piece_id=HCWHTXYBAT2S76PVNDAW7VZD2G5BOVRVVI7S57KO52VE2PYYUBVA, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.204616819 (555h8m48.075699854s) [noise=true , address=145.79.227.29:4942 , piece_id=32IV2553K5LM4IBPODY4PC36VY4AWDWYMBGJC6S3VC54K5DRUQZA, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.204884868 (555h8m48.075967903s) [noise=true , address=145.79.227.2:4113 , piece_id=LG6UO4VALLB4IMLBZSE44ST3SZUYMVF6XMQ3QDS5I752INHUXO4A, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.205146527 (555h8m48.076229562s) [noise=true , address=45.91.116.10:4081 , piece_id=SPBBD6TRT5KLITNFB4THJS5JGALVCIGGNBJNF57XIPPAQ4VWTX6A, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.205410911 (555h8m48.076493946s) [noise=true , address=145.79.227.9:4366 , piece_id=7VDYG6UIXNXBD636HAM253FHQKOC5W32GHFLQ3CNSR2JZSF3T6MQ, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.205723767 (555h8m48.076806812s) [noise=true , address=145.79.227.46:4602 , piece_id=MUBSLMB7UXCE23B6RB7ZTVXRXT4MHKZZ3R4X23BKCCC5NNDKD2HQ, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.206052121 (555h8m48.077135156s) [noise=true , address=145.79.227.3:4151 , piece_id=DZYAVTVZR6KUYTBIGT3ZOU3IHP6AZ7K4A7TYQR3EN7LSGSQLIL3Q, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.206368544 (555h8m48.077451589s) [noise=true , address=145.79.227.50:4374 , piece_id=FREM4FLXUJVAWIRRBB5NE4YSXKJ6QIUDVKHTCMLUEEAKPAKSJSVQ, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.206793653 (555h8m48.077876688s) [noise=true , address=145.79.227.7:4295 , piece_id=S7V6JQCIWML56ZHSI6Y7FUE7H4L5HOYI3ZB3KECQ4MQHEN6HAE2Q, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.207779020 (555h8m48.078862055s) [noise=true , address=145.79.227.19:4078 , piece_id=42ESUDAV7R45YMJGTWYFVFTW36AOJN42DXL6JYV75BS7AETRPP6A, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.208788222 (555h8m48.079871257s) [noise=true , address=145.79.227.30:4761 , piece_id=NXTV5NYP7NCS3CKKZLHNCEBKVOWGMWNE7S7ONXVZUVR6TXL3IBLQ, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.209870072 (555h8m48.080953107s) [noise=true , address=45.91.116.14:4191 , piece_id=UIPUC3NJRS4S6BAASNOF7E4LUAMJFCJUT76FRPAMWJH6PXD344VA, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.211042934 (555h8m48.082125969s) [noise=true , address=45.91.116.6:4975 , piece_id=3B24J7HKP323UDIOCACO5BLGP6BLDJZLF76IQQRBNL4SJ6T62HEA, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:49.825604830 (555h8m49.696687875s) [upload=486703]: Next returned nil segment Dec 28 18:37:49.825713468 (555h8m49.696796533s) [upload=486703]: Waiting for error group managing segments... Dec 28 18:37:50.202777853 (555h8m50.073860898s) [noise=true , address=145.79.227.46:4602 , piece_id=MUBSLMB7UXCE23B6RB7ZTVXRXT4MHKZZ3R4X23BKCCC5NNDKD2HQ, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.213864273 (555h8m50.084947308s) [noise=true , address=145.79.227.3:4151 , piece_id=DZYAVTVZR6KUYTBIGT3ZOU3IHP6AZ7K4A7TYQR3EN7LSGSQLIL3Q, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.214459827 (555h8m50.085542872s) [noise=true , address=145.79.227.2:4113 , piece_id=LG6UO4VALLB4IMLBZSE44ST3SZUYMVF6XMQ3QDS5I752INHUXO4A, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.223913449 (555h8m50.094996484s) [noise=true , address=145.79.227.19:4078 , piece_id=42ESUDAV7R45YMJGTWYFVFTW36AOJN42DXL6JYV75BS7AETRPP6A, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.226410564 (555h8m50.097493609s) [noise=true , address=145.79.227.7:4295 , piece_id=S7V6JQCIWML56ZHSI6Y7FUE7H4L5HOYI3ZB3KECQ4MQHEN6HAE2Q, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.227179488 (555h8m50.098262523s) [noise=true , address=145.79.227.9:4366 , piece_id=7VDYG6UIXNXBD636HAM253FHQKOC5W32GHFLQ3CNSR2JZSF3T6MQ, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.231591910 (555h8m50.102674965s) [noise=true , address=145.79.227.36:4519 , piece_id=LHQKDQ6XHMOZFOU66MMMLKBF2W5R6WY7UQ6WLAUKPQOM4KNYFGGQ, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.233960269 (555h8m50.105043314s) [noise=true , address=145.79.227.50:4374 , piece_id=FREM4FLXUJVAWIRRBB5NE4YSXKJ6QIUDVKHTCMLUEEAKPAKSJSVQ, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.269409902 (555h8m50.140492937s) [noise=true , address=145.79.227.16:4911 , piece_id=HCWHTXYBAT2S76PVNDAW7VZD2G5BOVRVVI7S57KO52VE2PYYUBVA, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.475175736 (555h8m50.346258771s) [noise=true , address=77.90.31.11:4556 , piece_id=JZTPKCAMECPIJKWWXJCANZRXUCMIDRELAUXZTF537ETNSBOJMRMQ, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.499048384 (555h8m50.370131419s) [noise=true , address=45.91.116.14:4191 , piece_id=UIPUC3NJRS4S6BAASNOF7E4LUAMJFCJUT76FRPAMWJH6PXD344VA, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.754699638 (555h8m50.625782673s) [noise=true , address=45.91.116.5:4298 , piece_id=O45R525ETFGWT43PDWDUJV5VYPIKXUDHDHR4JKAKPOWZ6BRUGULA, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.799090025 (555h8m50.67017306s) [noise=true , address=45.91.116.10:4081 , piece_id=SPBBD6TRT5KLITNFB4THJS5JGALVCIGGNBJNF57XIPPAQ4VWTX6A, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.799110284 (555h8m50.670193319s) [seg_pos={0 0}, upload=486703]: Segment reached optimal threshold of 13 pieces. Dec 28 18:37:50.799322979 (555h8m50.670406014s) [noise=true , address=145.79.227.29:4942 , piece_id=32IV2553K5LM4IBPODY4PC36VY4AWDWYMBGJC6S3VC54K5DRUQZA, seg_pos={0 0}, upload=486703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 28 18:37:50.799332867 (555h8m50.670415942s) [noise=true , address=145.79.227.29:4942 , piece_id=32IV2553K5LM4IBPODY4PC36VY4AWDWYMBGJC6S3VC54K5DRUQZA, seg_pos={0 0}, upload=486703]: Long tail cancellation due to optimal threshold Dec 28 18:37:50.799460621 (555h8m50.670543656s) [noise=true , address=145.79.227.30:4761 , piece_id=NXTV5NYP7NCS3CKKZLHNCEBKVOWGMWNE7S7ONXVZUVR6TXL3IBLQ, seg_pos={0 0}, upload=486703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 18:37:50.799466852 (555h8m50.670549887s) [noise=true , address=145.79.227.30:4761 , piece_id=NXTV5NYP7NCS3CKKZLHNCEBKVOWGMWNE7S7ONXVZUVR6TXL3IBLQ, seg_pos={0 0}, upload=486703]: Long tail cancellation due to optimal threshold Dec 28 18:37:50.799714804 (555h8m50.670797869s) [noise=true , address=45.91.116.6:4975 , piece_id=3B24J7HKP323UDIOCACO5BLGP6BLDJZLF76IQQRBNL4SJ6T62HEA, seg_pos={0 0}, upload=486703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 28 18:37:50.799734803 (555h8m50.670817858s) [noise=true , address=45.91.116.6:4975 , piece_id=3B24J7HKP323UDIOCACO5BLGP6BLDJZLF76IQQRBNL4SJ6T62HEA, seg_pos={0 0}, upload=486703]: Long tail cancellation due to optimal threshold Dec 28 18:37:50.799762596 (555h8m50.670845631s) [seg_pos={0 0}, upload=486703]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:37:50.799889106 (555h8m50.670972151s) [upload=486703]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:37:50.799896490 (555h8m50.670979535s) [upload=486703]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:37:51.098664356 (555h8m50.969747401s) [upload=486703]: Done uploading segments... Dec 28 18:37:51.098690967 (555h8m50.969774002s) [upload=486703]: Upload finished. err: <nil> Duration: 30.002836231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.............X.X........................X............] duration: 34.010076533s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:38:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 34.01010568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 3.495097099s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:37:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.495125944s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...] duration: 14.336671485s trace: https://jaeger.dev.storj.io/trace/76f49c76a8e1da20 FAIL (2025-12-28 18:37:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: MELX67YAXCOOIXHLPA3HWG6EA2GRKUOM5QXVEIAJ7BS5PMEXD5WA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: FYEIMUXCYS6QIY3BRDU2IZAGXTDNOXO2ENVSKZ2YRL46PUKXSUIA) context deadline exceeded ======LOGS====== c 28 18:33:58.722754987 (555h4m58.593838022s) [upload=486649]: Uploading segments... Dec 28 18:33:58.722808368 (555h4m58.593891403s) [upload=486649]: Got next segment. Inline: false Dec 28 18:33:58.722877549 (555h4m58.593960584s) [upload=486649]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:33:58.722882610 (555h4m58.593965645s) [upload=486649]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:33:58.968658119 (555h4m58.839741164s) [seg_pos={0 0}, upload=486649]: Begin upload segment... Dec 28 18:33:58.968718123 (555h4m58.839801158s) [seg_pos={0 0}, upload=486649]: Done begin upload segment. Dec 28 18:33:58.968748762 (555h4m58.839831797s) [noise=true , address=77.90.31.19:4842 , piece_id=FYEIMUXCYS6QIY3BRDU2IZAGXTDNOXO2ENVSKZ2YRL46PUKXSUIA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.968781634 (555h4m58.839864679s) [noise=true , address=77.90.31.12:4590 , piece_id=MSFP5QZQGVNJSZL7EHTRABC2ROWUQ4L22UGGHF6LCNG33WKABWQQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.969214989 (555h4m58.840298034s) [noise=true , address=145.79.227.52:4446 , piece_id=AOEDOIZO2ZVTYHDAHFACYKORLRIGGNF36M5BMUBTML7GMGCYLB5Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.969636361 (555h4m58.840719396s) [noise=true , address=77.90.31.13:4629 , piece_id=MELX67YAXCOOIXHLPA3HWG6EA2GRKUOM5QXVEIAJ7BS5PMEXD5WA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.970158345 (555h4m58.84124139s) [noise=true , address=145.79.227.12:4765 , piece_id=OSCF4NIIBQOL4UFJTALLXNCWCCNJXA2E5C357BIR4AT6457TINSQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.970701309 (555h4m58.841784345s) [noise=true , address=145.79.227.6:4257 , piece_id=3JDEIUV6BCME6EUKHKT2LLRMPQCGRRB6MBSMJ3KVFD73LH737WMA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.971126038 (555h4m58.842209073s) [noise=true , address=145.79.227.13:4804 , piece_id=3ZKXUQNJSUOXL6MPEXDFPP4DOKM7JTLBWQH4MY6PUPPK6S5L34VQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.971570705 (555h4m58.84265374s) [noise=true , address=145.79.227.19:4078 , piece_id=VQRDD6ESLBAUNSOY3F4C36NEC5FDS4HWDQQ3HHE6QFYCN3DVTG5Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.972646433 (555h4m58.843729478s) [noise=true , address=145.79.227.47:4265 , piece_id=SG55NEYJEBCK2KBFZ7U3PLQOUK7P5DO6Z5DRQ2DUWX2QYUZJUYFA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.972911328 (555h4m58.843994373s) [noise=true , address=145.79.227.45:4231 , piece_id=J2DCYUTGYDQJ7FGVNGEWZQA2JNJ7IBV2NDGPFPUVBNYLM4QHMD5Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.973974452 (555h4m58.845057487s) [noise=true , address=77.90.31.18:4808 , piece_id=NHM3KRMS6EGNCCE26C6FBZEW3TDMP4FXN5WE47PY6ODRE27LRLKQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.975212530 (555h4m58.846295565s) [noise=true , address=145.79.227.21:4475 , piece_id=GGBKTA4ICTXO3BWP5MYKWKOWMSCMKRVDWGV2TNWWBNOYCARKF35Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.978087724 (555h4m58.849170769s) [noise=true , address=145.79.227.7:4294 , piece_id=AOGDE3GEOSHGLZBF764HP6TFYEHRSGG4L2BFOQFUJE7UWOOAVRJA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.981692148 (555h4m58.852775193s) [noise=true , address=45.91.116.4:4939 , piece_id=RKDLUQITGFAHQV4AE2TCPTNCNAYARVZ25MF7JZLZZE4LPUQMVN3A, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.983781126 (555h4m58.854864161s) [noise=true , address=145.79.227.28:4727 , piece_id=RDEWPTFGENDP5VNTWQ36JS3VDB62UIZRO5T3EZYMKQ3PRXLNPIRQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.987752909 (555h4m58.858835954s) [noise=true , address=145.79.227.15:4875 , piece_id=KSZC4UZW5KMDXRSSRRKVWATYXERGXCJAK6MLPNXCW2F5FBLERONA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:34:00.690994828 (555h5m0.562077863s) [upload=486649]: Next returned nil segment Dec 28 18:34:00.691008914 (555h5m0.562091949s) [upload=486649]: Waiting for error group managing segments... Dec 28 18:34:00.977746422 (555h5m0.848829457s) [noise=true , address=145.79.227.12:4765 , piece_id=OSCF4NIIBQOL4UFJTALLXNCWCCNJXA2E5C357BIR4AT6457TINSQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.978844533 (555h5m0.849927568s) [noise=true , address=145.79.227.52:4446 , piece_id=AOEDOIZO2ZVTYHDAHFACYKORLRIGGNF36M5BMUBTML7GMGCYLB5Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.986731873 (555h5m0.857814918s) [noise=true , address=145.79.227.21:4475 , piece_id=GGBKTA4ICTXO3BWP5MYKWKOWMSCMKRVDWGV2TNWWBNOYCARKF35Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.987946485 (555h5m0.85902953s) [noise=true , address=145.79.227.15:4875 , piece_id=KSZC4UZW5KMDXRSSRRKVWATYXERGXCJAK6MLPNXCW2F5FBLERONA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.992659470 (555h5m0.863742505s) [noise=true , address=145.79.227.6:4257 , piece_id=3JDEIUV6BCME6EUKHKT2LLRMPQCGRRB6MBSMJ3KVFD73LH737WMA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.999582272 (555h5m0.870665317s) [noise=true , address=145.79.227.19:4078 , piece_id=VQRDD6ESLBAUNSOY3F4C36NEC5FDS4HWDQQ3HHE6QFYCN3DVTG5Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.005606773 (555h5m0.876689829s) [noise=true , address=145.79.227.45:4231 , piece_id=J2DCYUTGYDQJ7FGVNGEWZQA2JNJ7IBV2NDGPFPUVBNYLM4QHMD5Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.006067911 (555h5m0.877150956s) [noise=true , address=145.79.227.47:4265 , piece_id=SG55NEYJEBCK2KBFZ7U3PLQOUK7P5DO6Z5DRQ2DUWX2QYUZJUYFA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.011528650 (555h5m0.882611695s) [noise=true , address=145.79.227.28:4727 , piece_id=RDEWPTFGENDP5VNTWQ36JS3VDB62UIZRO5T3EZYMKQ3PRXLNPIRQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.049370617 (555h5m0.920453662s) [noise=true , address=145.79.227.13:4804 , piece_id=3ZKXUQNJSUOXL6MPEXDFPP4DOKM7JTLBWQH4MY6PUPPK6S5L34VQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.140450690 (555h5m1.011533755s) [noise=true , address=77.90.31.13:4629 , piece_id=MELX67YAXCOOIXHLPA3HWG6EA2GRKUOM5QXVEIAJ7BS5PMEXD5WA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.265885880 (555h5m1.136968935s) [noise=true , address=77.90.31.19:4842 , piece_id=FYEIMUXCYS6QIY3BRDU2IZAGXTDNOXO2ENVSKZ2YRL46PUKXSUIA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.266473128 (555h5m1.137556173s) [noise=true , address=77.90.31.12:4590 , piece_id=MSFP5QZQGVNJSZL7EHTRABC2ROWUQ4L22UGGHF6LCNG33WKABWQQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.266497515 (555h5m1.13758056s) [seg_pos={0 0}, upload=486649]: Segment reached optimal threshold of 13 pieces. Dec 28 18:34:01.267022615 (555h5m1.13810566s) [noise=true , address=77.90.31.18:4808 , piece_id=NHM3KRMS6EGNCCE26C6FBZEW3TDMP4FXN5WE47PY6ODRE27LRLKQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38010->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38010->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38010->77.90.31.18:4808: use of closed network connection Dec 28 18:34:01.267032714 (555h5m1.138115759s) [noise=true , address=77.90.31.18:4808 , piece_id=NHM3KRMS6EGNCCE26C6FBZEW3TDMP4FXN5WE47PY6ODRE27LRLKQ, seg_pos={0 0}, upload=486649]: Long tail cancellation due to optimal threshold Dec 28 18:34:01.267046881 (555h5m1.138129926s) [noise=true , address=45.91.116.4:4939 , piece_id=RKDLUQITGFAHQV4AE2TCPTNCNAYARVZ25MF7JZLZZE4LPUQMVN3A, seg_pos={0 0}, upload=486649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 18:34:01.267059134 (555h5m1.138142179s) [noise=true , address=45.91.116.4:4939 , piece_id=RKDLUQITGFAHQV4AE2TCPTNCNAYARVZ25MF7JZLZZE4LPUQMVN3A, seg_pos={0 0}, upload=486649]: Long tail cancellation due to optimal threshold Dec 28 18:34:01.267229859 (555h5m1.138312904s) [noise=true , address=145.79.227.7:4294 , piece_id=AOGDE3GEOSHGLZBF764HP6TFYEHRSGG4L2BFOQFUJE7UWOOAVRJA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:49112->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:49112->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->145.79.227.7:4294: use of closed network connection Dec 28 18:34:01.267238606 (555h5m1.138321661s) [noise=true , address=145.79.227.7:4294 , piece_id=AOGDE3GEOSHGLZBF764HP6TFYEHRSGG4L2BFOQFUJE7UWOOAVRJA, seg_pos={0 0}, upload=486649]: Long tail cancellation due to optimal threshold Dec 28 18:34:01.267260468 (555h5m1.138343513s) [seg_pos={0 0}, upload=486649]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:34:01.267375306 (555h5m1.138458351s) [upload=486649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:34:01.267383311 (555h5m1.138466356s) [upload=486649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:34:01.563507573 (555h5m1.434590618s) [upload=486649]: Done uploading segments... Dec 28 18:34:01.563533192 (555h5m1.434616237s) [upload=486649]: Upload finished. err: <nil> Duration: 14.336702553s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)