Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.853699663s FAIL (2025-10-25 17:37:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.853736923s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.912472355s FAIL (2025-10-25 17:36:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.912510618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 14.432499941s FAIL (2025-10-25 17:35:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 14.43254067s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.899617243s FAIL (2025-10-25 17:34:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.899647602s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.801080198s FAIL (2025-10-25 17:33:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.801118689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.753091727s FAIL (2025-10-25 17:32:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.75313547s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.647927863s FAIL (2025-10-25 17:31:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.647959162s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.937617541s FAIL (2025-10-25 17:31:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.937651626s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.207228229s FAIL (2025-10-25 17:30:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.207257713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.94491663s FAIL (2025-10-25 17:29:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.944943762s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.386151045s FAIL (2025-10-25 17:28:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.386195058s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.089791939s FAIL (2025-10-25 17:28:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.089838308s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 7.045706874s FAIL (2025-10-25 17:27:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 7.04576325s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.864826253s FAIL (2025-10-25 17:27:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.864866018s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.885300278s FAIL (2025-10-25 17:26:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.885326988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.944325224s FAIL (2025-10-25 17:25:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.944359489s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.956668237s FAIL (2025-10-25 17:25:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.956701581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.15613698s FAIL (2025-10-25 17:24:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.156170344s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.216110925s FAIL (2025-10-25 17:23:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.216143687s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.975378444s FAIL (2025-10-25 17:22:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.975424422s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.954019973s FAIL (2025-10-25 17:22:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.954056492s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.920064416s FAIL (2025-10-25 17:21:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.920097229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.939649849s FAIL (2025-10-25 17:20:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.939684354s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.886457178s FAIL (2025-10-25 17:20:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.886499007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.851525193s FAIL (2025-10-25 17:19:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.851559098s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.9617219s FAIL (2025-10-25 17:18:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.961750886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.905788066s FAIL (2025-10-25 17:18:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.905823454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.897006268s FAIL (2025-10-25 17:17:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.8970494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.988431801s FAIL (2025-10-25 17:17:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.988496082s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.045079498s FAIL (2025-10-25 17:15:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.04511147s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.401269702s FAIL (2025-10-25 17:15:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.401309696s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.012667206s FAIL (2025-10-25 17:14:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.012698876s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.06142434s FAIL (2025-10-25 17:14:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.061459027s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.02016662s FAIL (2025-10-25 17:13:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.02019805s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 7.786510487s FAIL (2025-10-25 17:12:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 7.786548018s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.152005338s FAIL (2025-10-25 17:11:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.152144263s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.034411174s FAIL (2025-10-25 17:10:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.034432195s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.041065426s FAIL (2025-10-25 17:10:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.04109965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.135922313s FAIL (2025-10-25 17:10:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.135955405s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 8.459564776s FAIL (2025-10-25 17:09:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 8.459601615s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.982544567s FAIL (2025-10-25 17:08:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.982572399s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 7.610804248s FAIL (2025-10-25 17:07:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 7.610834366s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 7.384749502s FAIL (2025-10-25 17:06:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 7.384783177s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.80593948s FAIL (2025-10-25 17:05:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 5.805970048s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.082790928s FAIL (2025-10-25 17:04:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 6.08281289s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002921511s FAIL (2025-10-25 17:04:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 25 17:04:16.248306862 (771h33m1.710399346s) [upload=678603]: Starting upload Oct 25 17:04:16.248427753 (771h33m1.710520236s) [upload=678603]: Done starting upload Oct 25 17:04:16.248619708 (771h33m1.710712181s) [upload=678603]: Uploading segments... Oct 25 17:04:16.248718456 (771h33m1.710810929s) [upload=678603]: Got next segment. Inline: false Oct 25 17:04:16.249019019 (771h33m1.711111502s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 17:04:16.249028427 (771h33m1.71112091s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 17:04:18.290312809 (771h33m3.752405312s) [seg_pos={0 0}, upload=678603]: Begin upload segment... Oct 25 17:04:18.290397800 (771h33m3.752490304s) [seg_pos={0 0}, upload=678603]: Done begin upload segment. Oct 25 17:04:18.290432747 (771h33m3.75252525s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291120076 (771h33m3.753212589s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291308504 (771h33m3.753400987s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.291771175 (771h33m3.753863658s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292168112 (771h33m3.754260595s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292376408 (771h33m3.754468911s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.292579395 (771h33m3.754671878s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293082002 (771h33m3.755174485s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293182603 (771h33m3.755275096s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.293943322 (771h33m3.756035805s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.294421362 (771h33m3.756513845s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.295915558 (771h33m3.758008041s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.297452354 (771h33m3.759544827s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.298320196 (771h33m3.76041268s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299258724 (771h33m3.761351207s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:18.299810785 (771h33m3.761903288s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Uploading piece... Oct 25 17:04:19.922966929 (771h33m5.385059412s) [upload=678603]: Next returned nil segment Oct 25 17:04:19.922981487 (771h33m5.38507397s) [upload=678603]: Waiting for error group managing segments... Oct 25 17:04:20.308702214 (771h33m5.770794697s) [noise=true , address=145.79.227.25:4619 , piece_id=NXNHY7CSUWATUXIK6H6ISFRACJRBVX3266TDEE6VZKLIARM3VJQQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.312596289 (771h33m5.774688772s) [noise=true , address=145.79.227.20:4439 , piece_id=DXKFGY6TITJ6JBVULU2YTMKHHOR73QPVQMPKZVL7YNS7QJBVIBDA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.321257562 (771h33m5.783350045s) [noise=true , address=145.79.227.21:4474 , piece_id=C627EW6FI5LAN5SJJS6N6HX52MPRYK2VVAVOD6SKYGAKA42X6PGA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.353459892 (771h33m5.815552385s) [noise=true , address=145.79.227.26:4654 , piece_id=SLL73Z5TN7OFR2ZVBM4W4SQKZQRQK6BKCE4BYAIGXD67A47VC6LA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.357782512 (771h33m5.819874996s) [noise=true , address=145.79.227.11:4729 , piece_id=567ZJS6YW6BLOVUBXQ3WSWSEKUUMOWBIFBIJ7UVTYX3XFZDMWEBA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.358629296 (771h33m5.820721779s) [noise=true , address=145.79.227.14:4840 , piece_id=DZKGP45YH6XVONBIC2G77JK47O5GZHDI4BHZZA36QZGJOHHHNEWQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.361267309 (771h33m5.823359792s) [noise=true , address=145.79.227.13:4801 , piece_id=AYR4APF3NH37U2MKRXI54CZG5HH6FW2VLAH5LFVSOINPAPBQJQXA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.364692821 (771h33m5.826785304s) [noise=true , address=145.79.227.18:4007 , piece_id=RJ6RZBHOFVVYVKXNE5EVMDQ3MJTHAIQAWAR4PLM2CGP45EQLTQ3A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.373280564 (771h33m5.835373037s) [noise=true , address=145.79.227.2:4114 , piece_id=HCTLCUAS2VDMK6RRMYVURAOZDBAZ33ZRVXGGBLMLK23SMNCGHPHQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.376444679 (771h33m5.838537152s) [noise=true , address=145.79.227.22:4511 , piece_id=6325AQSB6O2RZFRGH5HO4OHKES2JA224LVHQZ3G63AAOCEPQ2Y2A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.387131890 (771h33m5.849224373s) [noise=true , address=145.79.227.9:4365 , piece_id=E6B5Z6O4QMQZ4FADOBTLVJU4KNPR3FYWREIDHZ2E56F53K4AHODA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.477743502 (771h33m5.939835985s) [noise=true , address=77.90.31.13:4627 , piece_id=HG6XDTP44UECHGZELRRH4G5ITECBMXTJ3BPKDED2SBGNA4OGPRLA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577020003 (771h33m6.039112487s) [noise=true , address=77.90.31.18:4807 , piece_id=QR6S7A6XAV7UZ5TGAQGABRCI3BLVNB7QQJHIZEGQAQOHJ22DJ6GA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: <nil> Oct 25 17:04:20.577054469 (771h33m6.039146942s) [seg_pos={0 0}, upload=678603]: Segment reached optimal threshold of 13 pieces. Oct 25 17:04:20.577184026 (771h33m6.039276499s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577191861 (771h33m6.039284334s) [noise=true , address=77.90.31.11:4555 , piece_id=RGTGGAXXNPKTM4CFQIH3WQ3XQOI32OW3ADYUSSG2OUQ737X5B5IA, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577211649 (771h33m6.039304142s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577223651 (771h33m6.039316135s) [noise=true , address=77.90.31.21:4879 , piece_id=A57TLJ3UJRDHMB4UC6BQTLIZCZON2GGAEYN7TWHDTKU3DN2XR25A, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577253028 (771h33m6.039345501s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 25 17:04:20.577259409 (771h33m6.039351893s) [noise=true , address=77.90.31.16:4735 , piece_id=GV2V3DCNUUQIRETPMKJWWWHVTOFD6IQCIRFAN6BFH7JZFVIMVEEQ, seg_pos={0 0}, upload=678603]: Long tail cancellation due to optimal threshold Oct 25 17:04:20.577274519 (771h33m6.039367002s) [seg_pos={0 0}, upload=678603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 17:04:20.577349832 (771h33m6.039442305s) [upload=678603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 17:04:20.577354882 (771h33m6.039447355s) [upload=678603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 17:04:21.682018356 (771h33m7.144110839s) [upload=678603]: Done uploading segments... Oct 25 17:04:21.682043664 (771h33m7.144136137s) [upload=678603]: Upload finished. err: <nil> Duration: 30.002950096s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.878840461s FAIL (2025-10-25 16:29:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 5.878873453s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 23.342213084s FAIL (2025-10-25 16:28:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 23.342282546s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.887461248s FAIL (2025-10-25 16:27:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 5.887510742s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 7.159932072s FAIL (2025-10-25 16:26:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 7.159965213s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)