Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.................X...................X...X.......X.X....] duration: 25.021018776s FAIL (2025-10-08 13:46:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 8 13:43:57.810063563 (360h12m43.272156046s) [upload=316904]: Starting upload Oct 8 13:43:57.810137954 (360h12m43.272230427s) [upload=316904]: Done starting upload Oct 8 13:43:57.810236702 (360h12m43.272329186s) [upload=316904]: Uploading segments... Oct 8 13:43:57.810342684 (360h12m43.272435167s) [upload=316904]: Got next segment. Inline: false Oct 8 13:43:57.810377731 (360h12m43.272470204s) [upload=316904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:43:57.810383904 (360h12m43.272476387s) [upload=316904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:43:58.063348919 (360h12m43.525441402s) [seg_pos={0 0}, upload=316904]: Begin upload segment... Oct 8 13:43:58.063432057 (360h12m43.52552454s) [seg_pos={0 0}, upload=316904]: Done begin upload segment. Oct 8 13:43:58.063454870 (360h12m43.525547353s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.063485028 (360h12m43.525577511s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.063542286 (360h12m43.525634769s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.063669789 (360h12m43.525762263s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.064020457 (360h12m43.52611294s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.064261206 (360h12m43.526353699s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.065543117 (360h12m43.5276356s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.066129293 (360h12m43.528221766s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.066674420 (360h12m43.528766904s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.071228953 (360h12m43.533321426s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.072519902 (360h12m43.534612385s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.074066978 (360h12m43.536159461s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.075496490 (360h12m43.537588963s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.077207959 (360h12m43.539300442s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.078484940 (360h12m43.540577424s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.078901363 (360h12m43.540993847s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:59.958821937 (360h12m45.42091442s) [upload=316904]: Next returned nil segment Oct 8 13:43:59.958841474 (360h12m45.420933957s) [upload=316904]: Waiting for error group managing segments... Oct 8 13:44:00.261248377 (360h12m45.72334087s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.348925963 (360h12m45.811018446s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.385044931 (360h12m45.847137414s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.392667596 (360h12m45.854760069s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.460578662 (360h12m45.922671135s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.534763896 (360h12m45.996856379s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.620447514 (360h12m46.082539997s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:01.006762044 (360h12m46.468854527s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:01.034533997 (360h12m46.49662647s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:01.191701205 (360h12m46.653793698s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:16.644858234 (360h13m2.106950717s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:26.364996329 (360h13m11.827088812s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:27.082525848 (360h13m12.544618321s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection Oct 8 13:44:27.082668189 (360h13m12.544760662s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection Oct 8 13:44:27.082730177 (360h13m12.54482266s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection Oct 8 13:44:27.082781424 (360h13m12.544873897s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection Oct 8 13:44:27.082802865 (360h13m12.544895338s) [seg_pos={0 0}, upload=316904]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 8 13:44:27.341447707 (360h13m12.80354021s) [upload=316904]: Done uploading segments... Oct 8 13:44:27.341473306 (360h13m12.803565789s) [upload=316904]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 25.021052241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X...X.................X...................X...X.......X.X...] duration: 10.274597976s FAIL (2025-10-08 13:45:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 8 13:43:57.810063563 (360h12m43.272156046s) [upload=316904]: Starting upload Oct 8 13:43:57.810137954 (360h12m43.272230427s) [upload=316904]: Done starting upload Oct 8 13:43:57.810236702 (360h12m43.272329186s) [upload=316904]: Uploading segments... Oct 8 13:43:57.810342684 (360h12m43.272435167s) [upload=316904]: Got next segment. Inline: false Oct 8 13:43:57.810377731 (360h12m43.272470204s) [upload=316904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:43:57.810383904 (360h12m43.272476387s) [upload=316904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:43:58.063348919 (360h12m43.525441402s) [seg_pos={0 0}, upload=316904]: Begin upload segment... Oct 8 13:43:58.063432057 (360h12m43.52552454s) [seg_pos={0 0}, upload=316904]: Done begin upload segment. Oct 8 13:43:58.063454870 (360h12m43.525547353s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.063485028 (360h12m43.525577511s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.063542286 (360h12m43.525634769s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.063669789 (360h12m43.525762263s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.064020457 (360h12m43.52611294s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.064261206 (360h12m43.526353699s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.065543117 (360h12m43.5276356s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.066129293 (360h12m43.528221766s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.066674420 (360h12m43.528766904s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.071228953 (360h12m43.533321426s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.072519902 (360h12m43.534612385s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.074066978 (360h12m43.536159461s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.075496490 (360h12m43.537588963s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.077207959 (360h12m43.539300442s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.078484940 (360h12m43.540577424s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.078901363 (360h12m43.540993847s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:59.958821937 (360h12m45.42091442s) [upload=316904]: Next returned nil segment Oct 8 13:43:59.958841474 (360h12m45.420933957s) [upload=316904]: Waiting for error group managing segments... Oct 8 13:44:00.261248377 (360h12m45.72334087s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.348925963 (360h12m45.811018446s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.385044931 (360h12m45.847137414s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.392667596 (360h12m45.854760069s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.460578662 (360h12m45.922671135s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.534763896 (360h12m45.996856379s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.620447514 (360h12m46.082539997s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:01.006762044 (360h12m46.468854527s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:01.034533997 (360h12m46.49662647s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:01.191701205 (360h12m46.653793698s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:16.644858234 (360h13m2.106950717s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:26.364996329 (360h13m11.827088812s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:27.082525848 (360h13m12.544618321s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection Oct 8 13:44:27.082668189 (360h13m12.544760662s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection Oct 8 13:44:27.082730177 (360h13m12.54482266s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection Oct 8 13:44:27.082781424 (360h13m12.544873897s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection Oct 8 13:44:27.082802865 (360h13m12.544895338s) [seg_pos={0 0}, upload=316904]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 8 13:44:27.341447707 (360h13m12.80354021s) [upload=316904]: Done uploading segments... Oct 8 13:44:27.341473306 (360h13m12.803565789s) [upload=316904]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.274619197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XX...X.................X...................X...X.......X.X..] duration: 6.776074239s FAIL (2025-10-08 13:45:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 8 13:43:57.810063563 (360h12m43.272156046s) [upload=316904]: Starting upload Oct 8 13:43:57.810137954 (360h12m43.272230427s) [upload=316904]: Done starting upload Oct 8 13:43:57.810236702 (360h12m43.272329186s) [upload=316904]: Uploading segments... Oct 8 13:43:57.810342684 (360h12m43.272435167s) [upload=316904]: Got next segment. Inline: false Oct 8 13:43:57.810377731 (360h12m43.272470204s) [upload=316904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:43:57.810383904 (360h12m43.272476387s) [upload=316904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:43:58.063348919 (360h12m43.525441402s) [seg_pos={0 0}, upload=316904]: Begin upload segment... Oct 8 13:43:58.063432057 (360h12m43.52552454s) [seg_pos={0 0}, upload=316904]: Done begin upload segment. Oct 8 13:43:58.063454870 (360h12m43.525547353s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.063485028 (360h12m43.525577511s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.063542286 (360h12m43.525634769s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.063669789 (360h12m43.525762263s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.064020457 (360h12m43.52611294s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.064261206 (360h12m43.526353699s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.065543117 (360h12m43.5276356s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.066129293 (360h12m43.528221766s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.066674420 (360h12m43.528766904s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.071228953 (360h12m43.533321426s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.072519902 (360h12m43.534612385s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.074066978 (360h12m43.536159461s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.075496490 (360h12m43.537588963s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.077207959 (360h12m43.539300442s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.078484940 (360h12m43.540577424s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.078901363 (360h12m43.540993847s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:59.958821937 (360h12m45.42091442s) [upload=316904]: Next returned nil segment Oct 8 13:43:59.958841474 (360h12m45.420933957s) [upload=316904]: Waiting for error group managing segments... Oct 8 13:44:00.261248377 (360h12m45.72334087s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.348925963 (360h12m45.811018446s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.385044931 (360h12m45.847137414s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.392667596 (360h12m45.854760069s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.460578662 (360h12m45.922671135s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.534763896 (360h12m45.996856379s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.620447514 (360h12m46.082539997s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:01.006762044 (360h12m46.468854527s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:01.034533997 (360h12m46.49662647s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:01.191701205 (360h12m46.653793698s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:16.644858234 (360h13m2.106950717s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:26.364996329 (360h13m11.827088812s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:27.082525848 (360h13m12.544618321s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection Oct 8 13:44:27.082668189 (360h13m12.544760662s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection Oct 8 13:44:27.082730177 (360h13m12.54482266s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection Oct 8 13:44:27.082781424 (360h13m12.544873897s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection Oct 8 13:44:27.082802865 (360h13m12.544895338s) [seg_pos={0 0}, upload=316904]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 8 13:44:27.341447707 (360h13m12.80354021s) [upload=316904]: Done uploading segments... Oct 8 13:44:27.341473306 (360h13m12.803565789s) [upload=316904]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.776171935s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.XX...X.................X...................X...X.......X.X.] duration: 21.826487883s FAIL (2025-10-08 13:44:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 8 13:43:57.810063563 (360h12m43.272156046s) [upload=316904]: Starting upload Oct 8 13:43:57.810137954 (360h12m43.272230427s) [upload=316904]: Done starting upload Oct 8 13:43:57.810236702 (360h12m43.272329186s) [upload=316904]: Uploading segments... Oct 8 13:43:57.810342684 (360h12m43.272435167s) [upload=316904]: Got next segment. Inline: false Oct 8 13:43:57.810377731 (360h12m43.272470204s) [upload=316904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:43:57.810383904 (360h12m43.272476387s) [upload=316904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:43:58.063348919 (360h12m43.525441402s) [seg_pos={0 0}, upload=316904]: Begin upload segment... Oct 8 13:43:58.063432057 (360h12m43.52552454s) [seg_pos={0 0}, upload=316904]: Done begin upload segment. Oct 8 13:43:58.063454870 (360h12m43.525547353s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.063485028 (360h12m43.525577511s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.063542286 (360h12m43.525634769s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.063669789 (360h12m43.525762263s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.064020457 (360h12m43.52611294s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.064261206 (360h12m43.526353699s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.065543117 (360h12m43.5276356s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.066129293 (360h12m43.528221766s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.066674420 (360h12m43.528766904s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.071228953 (360h12m43.533321426s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.072519902 (360h12m43.534612385s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.074066978 (360h12m43.536159461s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.075496490 (360h12m43.537588963s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.077207959 (360h12m43.539300442s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.078484940 (360h12m43.540577424s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.078901363 (360h12m43.540993847s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:59.958821937 (360h12m45.42091442s) [upload=316904]: Next returned nil segment Oct 8 13:43:59.958841474 (360h12m45.420933957s) [upload=316904]: Waiting for error group managing segments... Oct 8 13:44:00.261248377 (360h12m45.72334087s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.348925963 (360h12m45.811018446s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.385044931 (360h12m45.847137414s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.392667596 (360h12m45.854760069s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.460578662 (360h12m45.922671135s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.534763896 (360h12m45.996856379s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.620447514 (360h12m46.082539997s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:01.006762044 (360h12m46.468854527s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:01.034533997 (360h12m46.49662647s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:01.191701205 (360h12m46.653793698s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:16.644858234 (360h13m2.106950717s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:26.364996329 (360h13m11.827088812s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:27.082525848 (360h13m12.544618321s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection Oct 8 13:44:27.082668189 (360h13m12.544760662s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection Oct 8 13:44:27.082730177 (360h13m12.54482266s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection Oct 8 13:44:27.082781424 (360h13m12.544873897s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection Oct 8 13:44:27.082802865 (360h13m12.544895338s) [seg_pos={0 0}, upload=316904]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 8 13:44:27.341447707 (360h13m12.80354021s) [upload=316904]: Done uploading segments... Oct 8 13:44:27.341473306 (360h13m12.803565789s) [upload=316904]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 21.826510887s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..XX...X.................X...................X...X.......X.X] duration: 30.261245506s FAIL (2025-10-08 13:44:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 8 13:43:57.810063563 (360h12m43.272156046s) [upload=316904]: Starting upload Oct 8 13:43:57.810137954 (360h12m43.272230427s) [upload=316904]: Done starting upload Oct 8 13:43:57.810236702 (360h12m43.272329186s) [upload=316904]: Uploading segments... Oct 8 13:43:57.810342684 (360h12m43.272435167s) [upload=316904]: Got next segment. Inline: false Oct 8 13:43:57.810377731 (360h12m43.272470204s) [upload=316904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:43:57.810383904 (360h12m43.272476387s) [upload=316904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:43:58.063348919 (360h12m43.525441402s) [seg_pos={0 0}, upload=316904]: Begin upload segment... Oct 8 13:43:58.063432057 (360h12m43.52552454s) [seg_pos={0 0}, upload=316904]: Done begin upload segment. Oct 8 13:43:58.063454870 (360h12m43.525547353s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.063485028 (360h12m43.525577511s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.063542286 (360h12m43.525634769s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.063669789 (360h12m43.525762263s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.064020457 (360h12m43.52611294s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.064261206 (360h12m43.526353699s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.065543117 (360h12m43.5276356s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.066129293 (360h12m43.528221766s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.066674420 (360h12m43.528766904s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.071228953 (360h12m43.533321426s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.072519902 (360h12m43.534612385s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.074066978 (360h12m43.536159461s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.075496490 (360h12m43.537588963s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.077207959 (360h12m43.539300442s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.078484940 (360h12m43.540577424s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:58.078901363 (360h12m43.540993847s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Uploading piece... Oct 8 13:43:59.958821937 (360h12m45.42091442s) [upload=316904]: Next returned nil segment Oct 8 13:43:59.958841474 (360h12m45.420933957s) [upload=316904]: Waiting for error group managing segments... Oct 8 13:44:00.261248377 (360h12m45.72334087s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.348925963 (360h12m45.811018446s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.385044931 (360h12m45.847137414s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.392667596 (360h12m45.854760069s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.460578662 (360h12m45.922671135s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.534763896 (360h12m45.996856379s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:00.620447514 (360h12m46.082539997s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:01.006762044 (360h12m46.468854527s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:01.034533997 (360h12m46.49662647s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:01.191701205 (360h12m46.653793698s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:16.644858234 (360h13m2.106950717s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:26.364996329 (360h13m11.827088812s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil> Oct 8 13:44:27.082525848 (360h13m12.544618321s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection Oct 8 13:44:27.082668189 (360h13m12.544760662s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection Oct 8 13:44:27.082730177 (360h13m12.54482266s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection Oct 8 13:44:27.082781424 (360h13m12.544873897s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection Oct 8 13:44:27.082802865 (360h13m12.544895338s) [seg_pos={0 0}, upload=316904]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 8 13:44:27.341447707 (360h13m12.80354021s) [upload=316904]: Done uploading segments... Oct 8 13:44:27.341473306 (360h13m12.803565789s) [upload=316904]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.261273208s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..XX...X.................X...................X...X.......X.] duration: 13.800604836s FAIL (2025-10-08 13:42:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, Piece ID: XYMNGHWU64P6X6RQIYYV7CXT6PDQKACHZBKJRURGQBYBYPSGDWXQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: PYO2WHHXMZKBIPTVI6PK4WOXUYBIT2DWYGT2THZ5CEW3Q5KJRVZQ) context deadline exceeded; piecestore: (Node ID: 1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, Piece ID: DNK3R2OVA5CTUCT4THV2ILA2VCFHHERWDKBV35PF5YVPIQXCFECQ) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: UOM74L2PYK43DXSHEWQTXCO5Y62RCSSQDWPTRYOHS4SIHTXZFS2A) context deadline exceeded; piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: 5WTUFWJUZNCFJKT7IP2HAUL3CVLIZWDMWTU4GMXJDEVX5HF7GNNQ) context deadline exceeded; piecestore: (Node ID: 1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, Piece ID: GLWHDZFFFMIXEBUSS4VXCWF5ZABDAT4UL6XY2YIFEW2QVLLTUFNQ) context deadline exceeded; piecestore: (Node ID: 1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, Piece ID: 7CJVSL44WM4MZKBNIYXTADBKRO4RMVJDPTABYMJNOUJL6ZODTY3A) context deadline exceeded; piecestore: (Node ID: 1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, Piece ID: JZY2J5EJ3RMB6RRYQKBJV5FX3JDB7HJPIFJTBALAPRP442MF6PLQ) context deadline exceeded ======LOGS====== Oct 8 13:41:43.477091796 (360h10m28.939184269s) [upload=316873]: Starting upload Oct 8 13:41:43.477159455 (360h10m28.939251928s) [upload=316873]: Done starting upload Oct 8 13:41:43.477213859 (360h10m28.939306332s) [upload=316873]: Uploading segments... Oct 8 13:41:43.477339116 (360h10m28.9394316s) [upload=316873]: Got next segment. Inline: false Oct 8 13:41:43.477374774 (360h10m28.939467247s) [upload=316873]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:41:43.477379253 (360h10m28.939471726s) [upload=316873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:42:05.365500770 (360h10m50.827593263s) [seg_pos={0 0}, upload=316873]: Begin upload segment... Oct 8 13:42:05.365708946 (360h10m50.827801429s) [seg_pos={0 0}, upload=316873]: Done begin upload segment. Oct 8 13:42:05.365729495 (360h10m50.827821978s) [noise=true , address=45.91.116.5:4297 , piece_id=W3HAV4QGTODSRPCLXE4B7TJWSYCQY3VOADX4DKQLATYRAXMYSC3Q, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.365847720 (360h10m50.827940214s) [noise=true , address=145.79.227.15:4875 , piece_id=GLWHDZFFFMIXEBUSS4VXCWF5ZABDAT4UL6XY2YIFEW2QVLLTUFNQ, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.366911395 (360h10m50.829003878s) [noise=true , address=45.91.116.7:4009 , piece_id=K52L2KDZZMHXIYIWBGRPJL7LNVY5LTN4RKQFIU24MRLO3ESO3YZQ, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.367417419 (360h10m50.829509892s) [noise=true , address=145.79.227.7:4295 , piece_id=XYMNGHWU64P6X6RQIYYV7CXT6PDQKACHZBKJRURGQBYBYPSGDWXQ, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.367953840 (360h10m50.830046323s) [noise=true , address=77.90.31.10:4519 , piece_id=PYO2WHHXMZKBIPTVI6PK4WOXUYBIT2DWYGT2THZ5CEW3Q5KJRVZQ, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.368966689 (360h10m50.831059172s) [noise=true , address=145.79.227.12:4767 , piece_id=DNK3R2OVA5CTUCT4THV2ILA2VCFHHERWDKBV35PF5YVPIQXCFECQ, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.371254686 (360h10m50.833347159s) [noise=true , address=145.79.227.3:4150 , piece_id=TH3FSJ3JFBHED7ZXVIQTIQEOBV4Z4YRCRPUZVBFYKGCIOS7FBP6Q, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.372378766 (360h10m50.834471249s) [noise=true , address=145.79.227.19:4079 , piece_id=UOM74L2PYK43DXSHEWQTXCO5Y62RCSSQDWPTRYOHS4SIHTXZFS2A, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.372826138 (360h10m50.834918621s) [noise=true , address=45.91.116.9:4263 , piece_id=LQATXIS63RCPZN65Y5OCQPNNS4B4DWI673AWODWMFI5GOUPVFLHA, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.374370049 (360h10m50.836462532s) [noise=true , address=145.79.227.6:4257 , piece_id=AAF7QFGYKSK7X7S6BW7NFBPGFW3N5JBQFL53SZ7AZOPYY5XKA5TA, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.374551344 (360h10m50.836643827s) [noise=true , address=45.91.116.14:4189 , piece_id=3FDLLDM36J2VRMBG4VS7CJ7TPHWDJGNVESAXH2ESZYMX2FDRUUSQ, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.377712523 (360h10m50.839805026s) [noise=true , address=145.79.227.4:4185 , piece_id=5WTUFWJUZNCFJKT7IP2HAUL3CVLIZWDMWTU4GMXJDEVX5HF7GNNQ, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.383988624 (360h10m50.846081107s) [noise=true , address=45.91.116.13:4153 , piece_id=KRVH3542JSTBNMPVM23FTWNF6ZJZMX6HEDQHOWPNPFSY4QYDEHBA, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.389393596 (360h10m50.85148609s) [noise=true , address=145.79.227.5:4222 , piece_id=JZY2J5EJ3RMB6RRYQKBJV5FX3JDB7HJPIFJTBALAPRP442MF6PLQ, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.394115938 (360h10m50.856208421s) [noise=true , address=145.79.227.11:4731 , piece_id=7CJVSL44WM4MZKBNIYXTADBKRO4RMVJDPTABYMJNOUJL6ZODTY3A, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.397017834 (360h10m50.859110307s) [noise=true , address=45.91.116.10:4082 , piece_id=O3SV3QZBV42GX7L77YFYJD7PYM6WNLEVV5MADZFJRU4BEPAGBHUA, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:07.090305321 (360h10m52.552397804s) [upload=316873]: Next returned nil segment Oct 8 13:42:07.090395763 (360h10m52.552488256s) [upload=316873]: Waiting for error group managing segments... Oct 8 13:42:07.373463189 (360h10m52.835555673s) [noise=true , address=145.79.227.3:4150 , piece_id=TH3FSJ3JFBHED7ZXVIQTIQEOBV4Z4YRCRPUZVBFYKGCIOS7FBP6Q, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.388880707 (360h10m52.85097319s) [noise=true , address=145.79.227.19:4079 , piece_id=UOM74L2PYK43DXSHEWQTXCO5Y62RCSSQDWPTRYOHS4SIHTXZFS2A, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.390666929 (360h10m52.852759412s) [noise=true , address=145.79.227.5:4222 , piece_id=JZY2J5EJ3RMB6RRYQKBJV5FX3JDB7HJPIFJTBALAPRP442MF6PLQ, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.398421505 (360h10m52.860513978s) [noise=true , address=145.79.227.6:4257 , piece_id=AAF7QFGYKSK7X7S6BW7NFBPGFW3N5JBQFL53SZ7AZOPYY5XKA5TA, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.399868250 (360h10m52.861960723s) [noise=true , address=145.79.227.7:4295 , piece_id=XYMNGHWU64P6X6RQIYYV7CXT6PDQKACHZBKJRURGQBYBYPSGDWXQ, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.402440969 (360h10m52.864533452s) [noise=true , address=145.79.227.12:4767 , piece_id=DNK3R2OVA5CTUCT4THV2ILA2VCFHHERWDKBV35PF5YVPIQXCFECQ, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.427474347 (360h10m52.889566831s) [noise=true , address=145.79.227.15:4875 , piece_id=GLWHDZFFFMIXEBUSS4VXCWF5ZABDAT4UL6XY2YIFEW2QVLLTUFNQ, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.435126549 (360h10m52.897219032s) [noise=true , address=145.79.227.4:4185 , piece_id=5WTUFWJUZNCFJKT7IP2HAUL3CVLIZWDMWTU4GMXJDEVX5HF7GNNQ, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.617554588 (360h10m53.079647071s) [noise=true , address=145.79.227.11:4731 , piece_id=7CJVSL44WM4MZKBNIYXTADBKRO4RMVJDPTABYMJNOUJL6ZODTY3A, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.619526633 (360h10m53.081619106s) [noise=true , address=45.91.116.9:4263 , piece_id=LQATXIS63RCPZN65Y5OCQPNNS4B4DWI673AWODWMFI5GOUPVFLHA, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.619792760 (360h10m53.081885243s) [noise=true , address=45.91.116.13:4153 , piece_id=KRVH3542JSTBNMPVM23FTWNF6ZJZMX6HEDQHOWPNPFSY4QYDEHBA, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.645473822 (360h10m53.107566325s) [noise=true , address=77.90.31.10:4519 , piece_id=PYO2WHHXMZKBIPTVI6PK4WOXUYBIT2DWYGT2THZ5CEW3Q5KJRVZQ, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.657350147 (360h10m53.11944262s) [noise=true , address=45.91.116.10:4082 , piece_id=O3SV3QZBV42GX7L77YFYJD7PYM6WNLEVV5MADZFJRU4BEPAGBHUA, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.657384573 (360h10m53.119477056s) [seg_pos={0 0}, upload=316873]: Segment reached optimal threshold of 13 pieces. Oct 8 13:42:07.657519760 (360h10m53.119612243s) [noise=true , address=45.91.116.5:4297 , piece_id=W3HAV4QGTODSRPCLXE4B7TJWSYCQY3VOADX4DKQLATYRAXMYSC3Q, seg_pos={0 0}, upload=316873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Oct 8 13:42:07.657529358 (360h10m53.119621841s) [noise=true , address=45.91.116.5:4297 , piece_id=W3HAV4QGTODSRPCLXE4B7TJWSYCQY3VOADX4DKQLATYRAXMYSC3Q, seg_pos={0 0}, upload=316873]: Long tail cancellation due to optimal threshold Oct 8 13:42:07.657548564 (360h10m53.119641067s) [noise=true , address=45.91.116.14:4189 , piece_id=3FDLLDM36J2VRMBG4VS7CJ7TPHWDJGNVESAXH2ESZYMX2FDRUUSQ, seg_pos={0 0}, upload=316873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Oct 8 13:42:07.657561149 (360h10m53.119653632s) [noise=true , address=45.91.116.14:4189 , piece_id=3FDLLDM36J2VRMBG4VS7CJ7TPHWDJGNVESAXH2ESZYMX2FDRUUSQ, seg_pos={0 0}, upload=316873]: Long tail cancellation due to optimal threshold Oct 8 13:42:07.657659616 (360h10m53.1197521s) [noise=true , address=45.91.116.7:4009 , piece_id=K52L2KDZZMHXIYIWBGRPJL7LNVY5LTN4RKQFIU24MRLO3ESO3YZQ, seg_pos={0 0}, upload=316873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:42730->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:42730->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42730->45.91.116.7:4009: use of closed network connection Oct 8 13:42:07.657669355 (360h10m53.119761838s) [noise=true , address=45.91.116.7:4009 , piece_id=K52L2KDZZMHXIYIWBGRPJL7LNVY5LTN4RKQFIU24MRLO3ESO3YZQ, seg_pos={0 0}, upload=316873]: Long tail cancellation due to optimal threshold Oct 8 13:42:07.657697509 (360h10m53.119790002s) [seg_pos={0 0}, upload=316873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:42:07.657786067 (360h10m53.11987855s) [upload=316873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:42:07.657863684 (360h10m53.119956177s) [upload=316873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:42:07.993954027 (360h10m53.45604651s) [upload=316873]: Done uploading segments... Oct 8 13:42:07.993978625 (360h10m53.456071118s) [upload=316873]: Upload finished. err: <nil> Duration: 13.800636197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX..XX...X.................X...................X...X.......X] duration: 30.002139963s FAIL (2025-10-08 13:42:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, Piece ID: XYMNGHWU64P6X6RQIYYV7CXT6PDQKACHZBKJRURGQBYBYPSGDWXQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: PYO2WHHXMZKBIPTVI6PK4WOXUYBIT2DWYGT2THZ5CEW3Q5KJRVZQ) context deadline exceeded; piecestore: (Node ID: 1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, Piece ID: DNK3R2OVA5CTUCT4THV2ILA2VCFHHERWDKBV35PF5YVPIQXCFECQ) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: UOM74L2PYK43DXSHEWQTXCO5Y62RCSSQDWPTRYOHS4SIHTXZFS2A) context deadline exceeded; piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: 5WTUFWJUZNCFJKT7IP2HAUL3CVLIZWDMWTU4GMXJDEVX5HF7GNNQ) context deadline exceeded; piecestore: (Node ID: 1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, Piece ID: GLWHDZFFFMIXEBUSS4VXCWF5ZABDAT4UL6XY2YIFEW2QVLLTUFNQ) context deadline exceeded; piecestore: (Node ID: 1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, Piece ID: 7CJVSL44WM4MZKBNIYXTADBKRO4RMVJDPTABYMJNOUJL6ZODTY3A) context deadline exceeded; piecestore: (Node ID: 1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, Piece ID: JZY2J5EJ3RMB6RRYQKBJV5FX3JDB7HJPIFJTBALAPRP442MF6PLQ) context deadline exceeded ======LOGS====== Oct 8 13:41:43.477091796 (360h10m28.939184269s) [upload=316873]: Starting upload Oct 8 13:41:43.477159455 (360h10m28.939251928s) [upload=316873]: Done starting upload Oct 8 13:41:43.477213859 (360h10m28.939306332s) [upload=316873]: Uploading segments... Oct 8 13:41:43.477339116 (360h10m28.9394316s) [upload=316873]: Got next segment. Inline: false Oct 8 13:41:43.477374774 (360h10m28.939467247s) [upload=316873]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:41:43.477379253 (360h10m28.939471726s) [upload=316873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:42:05.365500770 (360h10m50.827593263s) [seg_pos={0 0}, upload=316873]: Begin upload segment... Oct 8 13:42:05.365708946 (360h10m50.827801429s) [seg_pos={0 0}, upload=316873]: Done begin upload segment. Oct 8 13:42:05.365729495 (360h10m50.827821978s) [noise=true , address=45.91.116.5:4297 , piece_id=W3HAV4QGTODSRPCLXE4B7TJWSYCQY3VOADX4DKQLATYRAXMYSC3Q, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.365847720 (360h10m50.827940214s) [noise=true , address=145.79.227.15:4875 , piece_id=GLWHDZFFFMIXEBUSS4VXCWF5ZABDAT4UL6XY2YIFEW2QVLLTUFNQ, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.366911395 (360h10m50.829003878s) [noise=true , address=45.91.116.7:4009 , piece_id=K52L2KDZZMHXIYIWBGRPJL7LNVY5LTN4RKQFIU24MRLO3ESO3YZQ, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.367417419 (360h10m50.829509892s) [noise=true , address=145.79.227.7:4295 , piece_id=XYMNGHWU64P6X6RQIYYV7CXT6PDQKACHZBKJRURGQBYBYPSGDWXQ, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.367953840 (360h10m50.830046323s) [noise=true , address=77.90.31.10:4519 , piece_id=PYO2WHHXMZKBIPTVI6PK4WOXUYBIT2DWYGT2THZ5CEW3Q5KJRVZQ, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.368966689 (360h10m50.831059172s) [noise=true , address=145.79.227.12:4767 , piece_id=DNK3R2OVA5CTUCT4THV2ILA2VCFHHERWDKBV35PF5YVPIQXCFECQ, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.371254686 (360h10m50.833347159s) [noise=true , address=145.79.227.3:4150 , piece_id=TH3FSJ3JFBHED7ZXVIQTIQEOBV4Z4YRCRPUZVBFYKGCIOS7FBP6Q, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.372378766 (360h10m50.834471249s) [noise=true , address=145.79.227.19:4079 , piece_id=UOM74L2PYK43DXSHEWQTXCO5Y62RCSSQDWPTRYOHS4SIHTXZFS2A, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.372826138 (360h10m50.834918621s) [noise=true , address=45.91.116.9:4263 , piece_id=LQATXIS63RCPZN65Y5OCQPNNS4B4DWI673AWODWMFI5GOUPVFLHA, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.374370049 (360h10m50.836462532s) [noise=true , address=145.79.227.6:4257 , piece_id=AAF7QFGYKSK7X7S6BW7NFBPGFW3N5JBQFL53SZ7AZOPYY5XKA5TA, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.374551344 (360h10m50.836643827s) [noise=true , address=45.91.116.14:4189 , piece_id=3FDLLDM36J2VRMBG4VS7CJ7TPHWDJGNVESAXH2ESZYMX2FDRUUSQ, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.377712523 (360h10m50.839805026s) [noise=true , address=145.79.227.4:4185 , piece_id=5WTUFWJUZNCFJKT7IP2HAUL3CVLIZWDMWTU4GMXJDEVX5HF7GNNQ, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.383988624 (360h10m50.846081107s) [noise=true , address=45.91.116.13:4153 , piece_id=KRVH3542JSTBNMPVM23FTWNF6ZJZMX6HEDQHOWPNPFSY4QYDEHBA, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.389393596 (360h10m50.85148609s) [noise=true , address=145.79.227.5:4222 , piece_id=JZY2J5EJ3RMB6RRYQKBJV5FX3JDB7HJPIFJTBALAPRP442MF6PLQ, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.394115938 (360h10m50.856208421s) [noise=true , address=145.79.227.11:4731 , piece_id=7CJVSL44WM4MZKBNIYXTADBKRO4RMVJDPTABYMJNOUJL6ZODTY3A, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:05.397017834 (360h10m50.859110307s) [noise=true , address=45.91.116.10:4082 , piece_id=O3SV3QZBV42GX7L77YFYJD7PYM6WNLEVV5MADZFJRU4BEPAGBHUA, seg_pos={0 0}, upload=316873]: Uploading piece... Oct 8 13:42:07.090305321 (360h10m52.552397804s) [upload=316873]: Next returned nil segment Oct 8 13:42:07.090395763 (360h10m52.552488256s) [upload=316873]: Waiting for error group managing segments... Oct 8 13:42:07.373463189 (360h10m52.835555673s) [noise=true , address=145.79.227.3:4150 , piece_id=TH3FSJ3JFBHED7ZXVIQTIQEOBV4Z4YRCRPUZVBFYKGCIOS7FBP6Q, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.388880707 (360h10m52.85097319s) [noise=true , address=145.79.227.19:4079 , piece_id=UOM74L2PYK43DXSHEWQTXCO5Y62RCSSQDWPTRYOHS4SIHTXZFS2A, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.390666929 (360h10m52.852759412s) [noise=true , address=145.79.227.5:4222 , piece_id=JZY2J5EJ3RMB6RRYQKBJV5FX3JDB7HJPIFJTBALAPRP442MF6PLQ, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.398421505 (360h10m52.860513978s) [noise=true , address=145.79.227.6:4257 , piece_id=AAF7QFGYKSK7X7S6BW7NFBPGFW3N5JBQFL53SZ7AZOPYY5XKA5TA, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.399868250 (360h10m52.861960723s) [noise=true , address=145.79.227.7:4295 , piece_id=XYMNGHWU64P6X6RQIYYV7CXT6PDQKACHZBKJRURGQBYBYPSGDWXQ, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.402440969 (360h10m52.864533452s) [noise=true , address=145.79.227.12:4767 , piece_id=DNK3R2OVA5CTUCT4THV2ILA2VCFHHERWDKBV35PF5YVPIQXCFECQ, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.427474347 (360h10m52.889566831s) [noise=true , address=145.79.227.15:4875 , piece_id=GLWHDZFFFMIXEBUSS4VXCWF5ZABDAT4UL6XY2YIFEW2QVLLTUFNQ, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.435126549 (360h10m52.897219032s) [noise=true , address=145.79.227.4:4185 , piece_id=5WTUFWJUZNCFJKT7IP2HAUL3CVLIZWDMWTU4GMXJDEVX5HF7GNNQ, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.617554588 (360h10m53.079647071s) [noise=true , address=145.79.227.11:4731 , piece_id=7CJVSL44WM4MZKBNIYXTADBKRO4RMVJDPTABYMJNOUJL6ZODTY3A, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.619526633 (360h10m53.081619106s) [noise=true , address=45.91.116.9:4263 , piece_id=LQATXIS63RCPZN65Y5OCQPNNS4B4DWI673AWODWMFI5GOUPVFLHA, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.619792760 (360h10m53.081885243s) [noise=true , address=45.91.116.13:4153 , piece_id=KRVH3542JSTBNMPVM23FTWNF6ZJZMX6HEDQHOWPNPFSY4QYDEHBA, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.645473822 (360h10m53.107566325s) [noise=true , address=77.90.31.10:4519 , piece_id=PYO2WHHXMZKBIPTVI6PK4WOXUYBIT2DWYGT2THZ5CEW3Q5KJRVZQ, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.657350147 (360h10m53.11944262s) [noise=true , address=45.91.116.10:4082 , piece_id=O3SV3QZBV42GX7L77YFYJD7PYM6WNLEVV5MADZFJRU4BEPAGBHUA, seg_pos={0 0}, upload=316873]: Done uploading piece. err: <nil> Oct 8 13:42:07.657384573 (360h10m53.119477056s) [seg_pos={0 0}, upload=316873]: Segment reached optimal threshold of 13 pieces. Oct 8 13:42:07.657519760 (360h10m53.119612243s) [noise=true , address=45.91.116.5:4297 , piece_id=W3HAV4QGTODSRPCLXE4B7TJWSYCQY3VOADX4DKQLATYRAXMYSC3Q, seg_pos={0 0}, upload=316873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Oct 8 13:42:07.657529358 (360h10m53.119621841s) [noise=true , address=45.91.116.5:4297 , piece_id=W3HAV4QGTODSRPCLXE4B7TJWSYCQY3VOADX4DKQLATYRAXMYSC3Q, seg_pos={0 0}, upload=316873]: Long tail cancellation due to optimal threshold Oct 8 13:42:07.657548564 (360h10m53.119641067s) [noise=true , address=45.91.116.14:4189 , piece_id=3FDLLDM36J2VRMBG4VS7CJ7TPHWDJGNVESAXH2ESZYMX2FDRUUSQ, seg_pos={0 0}, upload=316873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Oct 8 13:42:07.657561149 (360h10m53.119653632s) [noise=true , address=45.91.116.14:4189 , piece_id=3FDLLDM36J2VRMBG4VS7CJ7TPHWDJGNVESAXH2ESZYMX2FDRUUSQ, seg_pos={0 0}, upload=316873]: Long tail cancellation due to optimal threshold Oct 8 13:42:07.657659616 (360h10m53.1197521s) [noise=true , address=45.91.116.7:4009 , piece_id=K52L2KDZZMHXIYIWBGRPJL7LNVY5LTN4RKQFIU24MRLO3ESO3YZQ, seg_pos={0 0}, upload=316873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:42730->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:42730->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42730->45.91.116.7:4009: use of closed network connection Oct 8 13:42:07.657669355 (360h10m53.119761838s) [noise=true , address=45.91.116.7:4009 , piece_id=K52L2KDZZMHXIYIWBGRPJL7LNVY5LTN4RKQFIU24MRLO3ESO3YZQ, seg_pos={0 0}, upload=316873]: Long tail cancellation due to optimal threshold Oct 8 13:42:07.657697509 (360h10m53.119790002s) [seg_pos={0 0}, upload=316873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:42:07.657786067 (360h10m53.11987855s) [upload=316873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:42:07.657863684 (360h10m53.119956177s) [upload=316873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:42:07.993954027 (360h10m53.45604651s) [upload=316873]: Done uploading segments... Oct 8 13:42:07.993978625 (360h10m53.456071118s) [upload=316873]: Upload finished. err: <nil> Duration: 30.002169129s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.XX..XX...X.................X...................X...X.......] duration: 6.229476204s FAIL (2025-10-08 13:41:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 13:34:49.838828743 (360h3m35.300921216s) [upload=316776]: Starting upload Oct 8 13:34:49.838915268 (360h3m35.301007741s) [upload=316776]: Done starting upload Oct 8 13:34:49.839094489 (360h3m35.301186962s) [upload=316776]: Uploading segments... Oct 8 13:34:49.839156237 (360h3m35.30124872s) [upload=316776]: Got next segment. Inline: false Oct 8 13:34:49.839191444 (360h3m35.301283917s) [upload=316776]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:34:49.839196824 (360h3m35.301289297s) [upload=316776]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:34:50.135260881 (360h3m35.597353354s) [seg_pos={0 0}, upload=316776]: Begin upload segment... Oct 8 13:34:50.135320996 (360h3m35.597413469s) [seg_pos={0 0}, upload=316776]: Done begin upload segment. Oct 8 13:34:50.135358337 (360h3m35.59745081s) [noise=true , address=77.90.31.21:4879 , piece_id=NQQACZC75N2OVZEYYCSGOJRSDNN7I4MM6FK2S244OOTC3FNB6APQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135365360 (360h3m35.597457843s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135441725 (360h3m35.597534198s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5EEIL6ATKQFJOYC4EFFGS2XTDN5OI6XXFEHRX2S222LEOHCQ4UQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135879640 (360h3m35.597972123s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.136377998 (360h3m35.598470481s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QY4YQ675MWT2RZPWNZ5AHX3KD22YBDTD2OTDQPT2EEUNTOPOTQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.136750097 (360h3m35.59884257s) [noise=true , address=45.91.116.14:4191 , piece_id=6GSU5AITXXNDAPZFGXXSEG4ELSD6YWIT5NJDOH2UOG4W2Z6WKA5A, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.137145740 (360h3m35.599238223s) [noise=true , address=145.79.227.13:4801 , piece_id=YLRCBTEMK7OL5B2F6L7OJGE6547YXBWPYSVHQRNLKTWC4G2TAN2A, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.138479160 (360h3m35.600571633s) [noise=true , address=77.90.31.11:4554 , piece_id=ZTJHP7NP4JSUTEFVYH6BETGYXZTVBSL3UYZL6ACD6U44KHQLTUNA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.139189071 (360h3m35.601281544s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.139862864 (360h3m35.601955347s) [noise=true , address=45.91.116.12:4226 , piece_id=A57OSOCQPWEBVNYNUPRIR3CHOSBHL2BMO4TM3QRM37D4MIUUQYJA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.140477925 (360h3m35.602570408s) [noise=true , address=45.91.116.9:4263 , piece_id=NS4BJHEUPAKAY2DBCP2CYH5BIADR6WGS3YBOADC7RSQWHNFD5JMA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.141060986 (360h3m35.603153469s) [noise=true , address=77.90.31.12:4592 , piece_id=UJWSOJP73TKIWPZL2PTUVRLYLSG75X7WS5GFTTKP63B5MXNIA5MQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.141495383 (360h3m35.603587866s) [noise=true , address=45.91.116.11:4119 , piece_id=4KGVJZZ5OSH6JUANPHHLRGYTAQECJAWP3HHTALP27XMSL57BB5EQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.142988015 (360h3m35.605080488s) [noise=true , address=45.91.116.6:4974 , piece_id=YUN4QEGIC4XY2AVGLJHEOJQG27JKCERNO6XTVEHGXT3K25TG24EQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.144130370 (360h3m35.606222853s) [noise=true , address=145.79.227.3:4151 , piece_id=XMJBO3O2BQHP5CDRGIKCMFCSQUXWP2FWEF7A3YKXTHSP4NR6Y6QQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.145590350 (360h3m35.607682834s) [noise=true , address=77.90.31.17:4770 , piece_id=4YUB24NXX4TCQSXVGHOWPQBDSX3PGZJJ2G2WIIVYOJVYISDVUKHQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:51.846370751 (360h3m37.308463224s) [upload=316776]: Next returned nil segment Oct 8 13:34:51.846393184 (360h3m37.308485657s) [upload=316776]: Waiting for error group managing segments... Oct 8 13:34:52.125182896 (360h3m37.587275369s) [noise=true , address=145.79.227.13:4801 , piece_id=YLRCBTEMK7OL5B2F6L7OJGE6547YXBWPYSVHQRNLKTWC4G2TAN2A, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.142347141 (360h3m37.604439624s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5EEIL6ATKQFJOYC4EFFGS2XTDN5OI6XXFEHRX2S222LEOHCQ4UQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.145727207 (360h3m37.60781969s) [noise=true , address=145.79.227.3:4151 , piece_id=XMJBO3O2BQHP5CDRGIKCMFCSQUXWP2FWEF7A3YKXTHSP4NR6Y6QQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.321404612 (360h3m37.783497085s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QY4YQ675MWT2RZPWNZ5AHX3KD22YBDTD2OTDQPT2EEUNTOPOTQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.392060135 (360h3m37.854152618s) [noise=true , address=45.91.116.11:4119 , piece_id=4KGVJZZ5OSH6JUANPHHLRGYTAQECJAWP3HHTALP27XMSL57BB5EQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.401936852 (360h3m37.864029335s) [noise=true , address=45.91.116.6:4974 , piece_id=YUN4QEGIC4XY2AVGLJHEOJQG27JKCERNO6XTVEHGXT3K25TG24EQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.408003444 (360h3m37.870095927s) [noise=true , address=77.90.31.11:4554 , piece_id=ZTJHP7NP4JSUTEFVYH6BETGYXZTVBSL3UYZL6ACD6U44KHQLTUNA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.411284171 (360h3m37.873376655s) [noise=true , address=77.90.31.17:4770 , piece_id=4YUB24NXX4TCQSXVGHOWPQBDSX3PGZJJ2G2WIIVYOJVYISDVUKHQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.416591808 (360h3m37.878684291s) [noise=true , address=45.91.116.9:4263 , piece_id=NS4BJHEUPAKAY2DBCP2CYH5BIADR6WGS3YBOADC7RSQWHNFD5JMA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.418720061 (360h3m37.880812544s) [noise=true , address=77.90.31.21:4879 , piece_id=NQQACZC75N2OVZEYYCSGOJRSDNN7I4MM6FK2S244OOTC3FNB6APQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.502015021 (360h3m37.964107504s) [noise=true , address=45.91.116.12:4226 , piece_id=A57OSOCQPWEBVNYNUPRIR3CHOSBHL2BMO4TM3QRM37D4MIUUQYJA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.503046987 (360h3m37.96513981s) [noise=true , address=45.91.116.14:4191 , piece_id=6GSU5AITXXNDAPZFGXXSEG4ELSD6YWIT5NJDOH2UOG4W2Z6WKA5A, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.526911169 (360h3m37.989003652s) [noise=true , address=77.90.31.12:4592 , piece_id=UJWSOJP73TKIWPZL2PTUVRLYLSG75X7WS5GFTTKP63B5MXNIA5MQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.526948269 (360h3m37.989040742s) [seg_pos={0 0}, upload=316776]: Segment reached optimal threshold of 13 pieces. Oct 8 13:34:52.527109416 (360h3m37.98920191s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 8 13:34:52.527120828 (360h3m37.989213311s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527176323 (360h3m37.989268806s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection Oct 8 13:34:52.527204477 (360h3m37.98929696s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Oct 8 13:34:52.527214617 (360h3m37.98930711s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527251387 (360h3m37.98934388s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527271655 (360h3m37.989364138s) [seg_pos={0 0}, upload=316776]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:34:52.527359282 (360h3m37.989451765s) [upload=316776]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:34:52.527366355 (360h3m37.989458848s) [upload=316776]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:34:52.880044686 (360h3m38.342137159s) [upload=316776]: Done uploading segments... Oct 8 13:34:52.880074503 (360h3m38.342166976s) [upload=316776]: Upload finished. err: <nil> Duration: 6.229509417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..XX..XX...X.................X...................X...X......] duration: 6.998608564s FAIL (2025-10-08 13:41:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 13:34:49.838828743 (360h3m35.300921216s) [upload=316776]: Starting upload Oct 8 13:34:49.838915268 (360h3m35.301007741s) [upload=316776]: Done starting upload Oct 8 13:34:49.839094489 (360h3m35.301186962s) [upload=316776]: Uploading segments... Oct 8 13:34:49.839156237 (360h3m35.30124872s) [upload=316776]: Got next segment. Inline: false Oct 8 13:34:49.839191444 (360h3m35.301283917s) [upload=316776]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:34:49.839196824 (360h3m35.301289297s) [upload=316776]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:34:50.135260881 (360h3m35.597353354s) [seg_pos={0 0}, upload=316776]: Begin upload segment... Oct 8 13:34:50.135320996 (360h3m35.597413469s) [seg_pos={0 0}, upload=316776]: Done begin upload segment. Oct 8 13:34:50.135358337 (360h3m35.59745081s) [noise=true , address=77.90.31.21:4879 , piece_id=NQQACZC75N2OVZEYYCSGOJRSDNN7I4MM6FK2S244OOTC3FNB6APQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135365360 (360h3m35.597457843s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135441725 (360h3m35.597534198s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5EEIL6ATKQFJOYC4EFFGS2XTDN5OI6XXFEHRX2S222LEOHCQ4UQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135879640 (360h3m35.597972123s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.136377998 (360h3m35.598470481s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QY4YQ675MWT2RZPWNZ5AHX3KD22YBDTD2OTDQPT2EEUNTOPOTQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.136750097 (360h3m35.59884257s) [noise=true , address=45.91.116.14:4191 , piece_id=6GSU5AITXXNDAPZFGXXSEG4ELSD6YWIT5NJDOH2UOG4W2Z6WKA5A, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.137145740 (360h3m35.599238223s) [noise=true , address=145.79.227.13:4801 , piece_id=YLRCBTEMK7OL5B2F6L7OJGE6547YXBWPYSVHQRNLKTWC4G2TAN2A, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.138479160 (360h3m35.600571633s) [noise=true , address=77.90.31.11:4554 , piece_id=ZTJHP7NP4JSUTEFVYH6BETGYXZTVBSL3UYZL6ACD6U44KHQLTUNA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.139189071 (360h3m35.601281544s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.139862864 (360h3m35.601955347s) [noise=true , address=45.91.116.12:4226 , piece_id=A57OSOCQPWEBVNYNUPRIR3CHOSBHL2BMO4TM3QRM37D4MIUUQYJA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.140477925 (360h3m35.602570408s) [noise=true , address=45.91.116.9:4263 , piece_id=NS4BJHEUPAKAY2DBCP2CYH5BIADR6WGS3YBOADC7RSQWHNFD5JMA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.141060986 (360h3m35.603153469s) [noise=true , address=77.90.31.12:4592 , piece_id=UJWSOJP73TKIWPZL2PTUVRLYLSG75X7WS5GFTTKP63B5MXNIA5MQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.141495383 (360h3m35.603587866s) [noise=true , address=45.91.116.11:4119 , piece_id=4KGVJZZ5OSH6JUANPHHLRGYTAQECJAWP3HHTALP27XMSL57BB5EQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.142988015 (360h3m35.605080488s) [noise=true , address=45.91.116.6:4974 , piece_id=YUN4QEGIC4XY2AVGLJHEOJQG27JKCERNO6XTVEHGXT3K25TG24EQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.144130370 (360h3m35.606222853s) [noise=true , address=145.79.227.3:4151 , piece_id=XMJBO3O2BQHP5CDRGIKCMFCSQUXWP2FWEF7A3YKXTHSP4NR6Y6QQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.145590350 (360h3m35.607682834s) [noise=true , address=77.90.31.17:4770 , piece_id=4YUB24NXX4TCQSXVGHOWPQBDSX3PGZJJ2G2WIIVYOJVYISDVUKHQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:51.846370751 (360h3m37.308463224s) [upload=316776]: Next returned nil segment Oct 8 13:34:51.846393184 (360h3m37.308485657s) [upload=316776]: Waiting for error group managing segments... Oct 8 13:34:52.125182896 (360h3m37.587275369s) [noise=true , address=145.79.227.13:4801 , piece_id=YLRCBTEMK7OL5B2F6L7OJGE6547YXBWPYSVHQRNLKTWC4G2TAN2A, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.142347141 (360h3m37.604439624s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5EEIL6ATKQFJOYC4EFFGS2XTDN5OI6XXFEHRX2S222LEOHCQ4UQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.145727207 (360h3m37.60781969s) [noise=true , address=145.79.227.3:4151 , piece_id=XMJBO3O2BQHP5CDRGIKCMFCSQUXWP2FWEF7A3YKXTHSP4NR6Y6QQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.321404612 (360h3m37.783497085s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QY4YQ675MWT2RZPWNZ5AHX3KD22YBDTD2OTDQPT2EEUNTOPOTQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.392060135 (360h3m37.854152618s) [noise=true , address=45.91.116.11:4119 , piece_id=4KGVJZZ5OSH6JUANPHHLRGYTAQECJAWP3HHTALP27XMSL57BB5EQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.401936852 (360h3m37.864029335s) [noise=true , address=45.91.116.6:4974 , piece_id=YUN4QEGIC4XY2AVGLJHEOJQG27JKCERNO6XTVEHGXT3K25TG24EQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.408003444 (360h3m37.870095927s) [noise=true , address=77.90.31.11:4554 , piece_id=ZTJHP7NP4JSUTEFVYH6BETGYXZTVBSL3UYZL6ACD6U44KHQLTUNA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.411284171 (360h3m37.873376655s) [noise=true , address=77.90.31.17:4770 , piece_id=4YUB24NXX4TCQSXVGHOWPQBDSX3PGZJJ2G2WIIVYOJVYISDVUKHQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.416591808 (360h3m37.878684291s) [noise=true , address=45.91.116.9:4263 , piece_id=NS4BJHEUPAKAY2DBCP2CYH5BIADR6WGS3YBOADC7RSQWHNFD5JMA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.418720061 (360h3m37.880812544s) [noise=true , address=77.90.31.21:4879 , piece_id=NQQACZC75N2OVZEYYCSGOJRSDNN7I4MM6FK2S244OOTC3FNB6APQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.502015021 (360h3m37.964107504s) [noise=true , address=45.91.116.12:4226 , piece_id=A57OSOCQPWEBVNYNUPRIR3CHOSBHL2BMO4TM3QRM37D4MIUUQYJA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.503046987 (360h3m37.96513981s) [noise=true , address=45.91.116.14:4191 , piece_id=6GSU5AITXXNDAPZFGXXSEG4ELSD6YWIT5NJDOH2UOG4W2Z6WKA5A, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.526911169 (360h3m37.989003652s) [noise=true , address=77.90.31.12:4592 , piece_id=UJWSOJP73TKIWPZL2PTUVRLYLSG75X7WS5GFTTKP63B5MXNIA5MQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.526948269 (360h3m37.989040742s) [seg_pos={0 0}, upload=316776]: Segment reached optimal threshold of 13 pieces. Oct 8 13:34:52.527109416 (360h3m37.98920191s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 8 13:34:52.527120828 (360h3m37.989213311s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527176323 (360h3m37.989268806s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection Oct 8 13:34:52.527204477 (360h3m37.98929696s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Oct 8 13:34:52.527214617 (360h3m37.98930711s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527251387 (360h3m37.98934388s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527271655 (360h3m37.989364138s) [seg_pos={0 0}, upload=316776]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:34:52.527359282 (360h3m37.989451765s) [upload=316776]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:34:52.527366355 (360h3m37.989458848s) [upload=316776]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:34:52.880044686 (360h3m38.342137159s) [upload=316776]: Done uploading segments... Oct 8 13:34:52.880074503 (360h3m38.342166976s) [upload=316776]: Upload finished. err: <nil> Duration: 6.998639735s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...XX..XX...X.................X...................X...X.....] duration: 20.745691819s FAIL (2025-10-08 13:40:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 13:34:49.838828743 (360h3m35.300921216s) [upload=316776]: Starting upload Oct 8 13:34:49.838915268 (360h3m35.301007741s) [upload=316776]: Done starting upload Oct 8 13:34:49.839094489 (360h3m35.301186962s) [upload=316776]: Uploading segments... Oct 8 13:34:49.839156237 (360h3m35.30124872s) [upload=316776]: Got next segment. Inline: false Oct 8 13:34:49.839191444 (360h3m35.301283917s) [upload=316776]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:34:49.839196824 (360h3m35.301289297s) [upload=316776]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:34:50.135260881 (360h3m35.597353354s) [seg_pos={0 0}, upload=316776]: Begin upload segment... Oct 8 13:34:50.135320996 (360h3m35.597413469s) [seg_pos={0 0}, upload=316776]: Done begin upload segment. Oct 8 13:34:50.135358337 (360h3m35.59745081s) [noise=true , address=77.90.31.21:4879 , piece_id=NQQACZC75N2OVZEYYCSGOJRSDNN7I4MM6FK2S244OOTC3FNB6APQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135365360 (360h3m35.597457843s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135441725 (360h3m35.597534198s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5EEIL6ATKQFJOYC4EFFGS2XTDN5OI6XXFEHRX2S222LEOHCQ4UQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135879640 (360h3m35.597972123s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.136377998 (360h3m35.598470481s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QY4YQ675MWT2RZPWNZ5AHX3KD22YBDTD2OTDQPT2EEUNTOPOTQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.136750097 (360h3m35.59884257s) [noise=true , address=45.91.116.14:4191 , piece_id=6GSU5AITXXNDAPZFGXXSEG4ELSD6YWIT5NJDOH2UOG4W2Z6WKA5A, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.137145740 (360h3m35.599238223s) [noise=true , address=145.79.227.13:4801 , piece_id=YLRCBTEMK7OL5B2F6L7OJGE6547YXBWPYSVHQRNLKTWC4G2TAN2A, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.138479160 (360h3m35.600571633s) [noise=true , address=77.90.31.11:4554 , piece_id=ZTJHP7NP4JSUTEFVYH6BETGYXZTVBSL3UYZL6ACD6U44KHQLTUNA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.139189071 (360h3m35.601281544s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.139862864 (360h3m35.601955347s) [noise=true , address=45.91.116.12:4226 , piece_id=A57OSOCQPWEBVNYNUPRIR3CHOSBHL2BMO4TM3QRM37D4MIUUQYJA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.140477925 (360h3m35.602570408s) [noise=true , address=45.91.116.9:4263 , piece_id=NS4BJHEUPAKAY2DBCP2CYH5BIADR6WGS3YBOADC7RSQWHNFD5JMA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.141060986 (360h3m35.603153469s) [noise=true , address=77.90.31.12:4592 , piece_id=UJWSOJP73TKIWPZL2PTUVRLYLSG75X7WS5GFTTKP63B5MXNIA5MQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.141495383 (360h3m35.603587866s) [noise=true , address=45.91.116.11:4119 , piece_id=4KGVJZZ5OSH6JUANPHHLRGYTAQECJAWP3HHTALP27XMSL57BB5EQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.142988015 (360h3m35.605080488s) [noise=true , address=45.91.116.6:4974 , piece_id=YUN4QEGIC4XY2AVGLJHEOJQG27JKCERNO6XTVEHGXT3K25TG24EQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.144130370 (360h3m35.606222853s) [noise=true , address=145.79.227.3:4151 , piece_id=XMJBO3O2BQHP5CDRGIKCMFCSQUXWP2FWEF7A3YKXTHSP4NR6Y6QQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.145590350 (360h3m35.607682834s) [noise=true , address=77.90.31.17:4770 , piece_id=4YUB24NXX4TCQSXVGHOWPQBDSX3PGZJJ2G2WIIVYOJVYISDVUKHQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:51.846370751 (360h3m37.308463224s) [upload=316776]: Next returned nil segment Oct 8 13:34:51.846393184 (360h3m37.308485657s) [upload=316776]: Waiting for error group managing segments... Oct 8 13:34:52.125182896 (360h3m37.587275369s) [noise=true , address=145.79.227.13:4801 , piece_id=YLRCBTEMK7OL5B2F6L7OJGE6547YXBWPYSVHQRNLKTWC4G2TAN2A, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.142347141 (360h3m37.604439624s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5EEIL6ATKQFJOYC4EFFGS2XTDN5OI6XXFEHRX2S222LEOHCQ4UQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.145727207 (360h3m37.60781969s) [noise=true , address=145.79.227.3:4151 , piece_id=XMJBO3O2BQHP5CDRGIKCMFCSQUXWP2FWEF7A3YKXTHSP4NR6Y6QQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.321404612 (360h3m37.783497085s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QY4YQ675MWT2RZPWNZ5AHX3KD22YBDTD2OTDQPT2EEUNTOPOTQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.392060135 (360h3m37.854152618s) [noise=true , address=45.91.116.11:4119 , piece_id=4KGVJZZ5OSH6JUANPHHLRGYTAQECJAWP3HHTALP27XMSL57BB5EQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.401936852 (360h3m37.864029335s) [noise=true , address=45.91.116.6:4974 , piece_id=YUN4QEGIC4XY2AVGLJHEOJQG27JKCERNO6XTVEHGXT3K25TG24EQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.408003444 (360h3m37.870095927s) [noise=true , address=77.90.31.11:4554 , piece_id=ZTJHP7NP4JSUTEFVYH6BETGYXZTVBSL3UYZL6ACD6U44KHQLTUNA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.411284171 (360h3m37.873376655s) [noise=true , address=77.90.31.17:4770 , piece_id=4YUB24NXX4TCQSXVGHOWPQBDSX3PGZJJ2G2WIIVYOJVYISDVUKHQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.416591808 (360h3m37.878684291s) [noise=true , address=45.91.116.9:4263 , piece_id=NS4BJHEUPAKAY2DBCP2CYH5BIADR6WGS3YBOADC7RSQWHNFD5JMA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.418720061 (360h3m37.880812544s) [noise=true , address=77.90.31.21:4879 , piece_id=NQQACZC75N2OVZEYYCSGOJRSDNN7I4MM6FK2S244OOTC3FNB6APQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.502015021 (360h3m37.964107504s) [noise=true , address=45.91.116.12:4226 , piece_id=A57OSOCQPWEBVNYNUPRIR3CHOSBHL2BMO4TM3QRM37D4MIUUQYJA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.503046987 (360h3m37.96513981s) [noise=true , address=45.91.116.14:4191 , piece_id=6GSU5AITXXNDAPZFGXXSEG4ELSD6YWIT5NJDOH2UOG4W2Z6WKA5A, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.526911169 (360h3m37.989003652s) [noise=true , address=77.90.31.12:4592 , piece_id=UJWSOJP73TKIWPZL2PTUVRLYLSG75X7WS5GFTTKP63B5MXNIA5MQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.526948269 (360h3m37.989040742s) [seg_pos={0 0}, upload=316776]: Segment reached optimal threshold of 13 pieces. Oct 8 13:34:52.527109416 (360h3m37.98920191s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 8 13:34:52.527120828 (360h3m37.989213311s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527176323 (360h3m37.989268806s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection Oct 8 13:34:52.527204477 (360h3m37.98929696s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Oct 8 13:34:52.527214617 (360h3m37.98930711s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527251387 (360h3m37.98934388s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527271655 (360h3m37.989364138s) [seg_pos={0 0}, upload=316776]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:34:52.527359282 (360h3m37.989451765s) [upload=316776]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:34:52.527366355 (360h3m37.989458848s) [upload=316776]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:34:52.880044686 (360h3m38.342137159s) [upload=316776]: Done uploading segments... Oct 8 13:34:52.880074503 (360h3m38.342166976s) [upload=316776]: Upload finished. err: <nil> Duration: 20.745728649s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....XX..XX...X.................X...................X...X....] duration: 25.568919245s FAIL (2025-10-08 13:39:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 13:34:49.838828743 (360h3m35.300921216s) [upload=316776]: Starting upload Oct 8 13:34:49.838915268 (360h3m35.301007741s) [upload=316776]: Done starting upload Oct 8 13:34:49.839094489 (360h3m35.301186962s) [upload=316776]: Uploading segments... Oct 8 13:34:49.839156237 (360h3m35.30124872s) [upload=316776]: Got next segment. Inline: false Oct 8 13:34:49.839191444 (360h3m35.301283917s) [upload=316776]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:34:49.839196824 (360h3m35.301289297s) [upload=316776]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:34:50.135260881 (360h3m35.597353354s) [seg_pos={0 0}, upload=316776]: Begin upload segment... Oct 8 13:34:50.135320996 (360h3m35.597413469s) [seg_pos={0 0}, upload=316776]: Done begin upload segment. Oct 8 13:34:50.135358337 (360h3m35.59745081s) [noise=true , address=77.90.31.21:4879 , piece_id=NQQACZC75N2OVZEYYCSGOJRSDNN7I4MM6FK2S244OOTC3FNB6APQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135365360 (360h3m35.597457843s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135441725 (360h3m35.597534198s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5EEIL6ATKQFJOYC4EFFGS2XTDN5OI6XXFEHRX2S222LEOHCQ4UQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135879640 (360h3m35.597972123s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.136377998 (360h3m35.598470481s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QY4YQ675MWT2RZPWNZ5AHX3KD22YBDTD2OTDQPT2EEUNTOPOTQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.136750097 (360h3m35.59884257s) [noise=true , address=45.91.116.14:4191 , piece_id=6GSU5AITXXNDAPZFGXXSEG4ELSD6YWIT5NJDOH2UOG4W2Z6WKA5A, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.137145740 (360h3m35.599238223s) [noise=true , address=145.79.227.13:4801 , piece_id=YLRCBTEMK7OL5B2F6L7OJGE6547YXBWPYSVHQRNLKTWC4G2TAN2A, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.138479160 (360h3m35.600571633s) [noise=true , address=77.90.31.11:4554 , piece_id=ZTJHP7NP4JSUTEFVYH6BETGYXZTVBSL3UYZL6ACD6U44KHQLTUNA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.139189071 (360h3m35.601281544s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.139862864 (360h3m35.601955347s) [noise=true , address=45.91.116.12:4226 , piece_id=A57OSOCQPWEBVNYNUPRIR3CHOSBHL2BMO4TM3QRM37D4MIUUQYJA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.140477925 (360h3m35.602570408s) [noise=true , address=45.91.116.9:4263 , piece_id=NS4BJHEUPAKAY2DBCP2CYH5BIADR6WGS3YBOADC7RSQWHNFD5JMA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.141060986 (360h3m35.603153469s) [noise=true , address=77.90.31.12:4592 , piece_id=UJWSOJP73TKIWPZL2PTUVRLYLSG75X7WS5GFTTKP63B5MXNIA5MQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.141495383 (360h3m35.603587866s) [noise=true , address=45.91.116.11:4119 , piece_id=4KGVJZZ5OSH6JUANPHHLRGYTAQECJAWP3HHTALP27XMSL57BB5EQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.142988015 (360h3m35.605080488s) [noise=true , address=45.91.116.6:4974 , piece_id=YUN4QEGIC4XY2AVGLJHEOJQG27JKCERNO6XTVEHGXT3K25TG24EQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.144130370 (360h3m35.606222853s) [noise=true , address=145.79.227.3:4151 , piece_id=XMJBO3O2BQHP5CDRGIKCMFCSQUXWP2FWEF7A3YKXTHSP4NR6Y6QQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.145590350 (360h3m35.607682834s) [noise=true , address=77.90.31.17:4770 , piece_id=4YUB24NXX4TCQSXVGHOWPQBDSX3PGZJJ2G2WIIVYOJVYISDVUKHQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:51.846370751 (360h3m37.308463224s) [upload=316776]: Next returned nil segment Oct 8 13:34:51.846393184 (360h3m37.308485657s) [upload=316776]: Waiting for error group managing segments... Oct 8 13:34:52.125182896 (360h3m37.587275369s) [noise=true , address=145.79.227.13:4801 , piece_id=YLRCBTEMK7OL5B2F6L7OJGE6547YXBWPYSVHQRNLKTWC4G2TAN2A, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.142347141 (360h3m37.604439624s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5EEIL6ATKQFJOYC4EFFGS2XTDN5OI6XXFEHRX2S222LEOHCQ4UQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.145727207 (360h3m37.60781969s) [noise=true , address=145.79.227.3:4151 , piece_id=XMJBO3O2BQHP5CDRGIKCMFCSQUXWP2FWEF7A3YKXTHSP4NR6Y6QQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.321404612 (360h3m37.783497085s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QY4YQ675MWT2RZPWNZ5AHX3KD22YBDTD2OTDQPT2EEUNTOPOTQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.392060135 (360h3m37.854152618s) [noise=true , address=45.91.116.11:4119 , piece_id=4KGVJZZ5OSH6JUANPHHLRGYTAQECJAWP3HHTALP27XMSL57BB5EQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.401936852 (360h3m37.864029335s) [noise=true , address=45.91.116.6:4974 , piece_id=YUN4QEGIC4XY2AVGLJHEOJQG27JKCERNO6XTVEHGXT3K25TG24EQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.408003444 (360h3m37.870095927s) [noise=true , address=77.90.31.11:4554 , piece_id=ZTJHP7NP4JSUTEFVYH6BETGYXZTVBSL3UYZL6ACD6U44KHQLTUNA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.411284171 (360h3m37.873376655s) [noise=true , address=77.90.31.17:4770 , piece_id=4YUB24NXX4TCQSXVGHOWPQBDSX3PGZJJ2G2WIIVYOJVYISDVUKHQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.416591808 (360h3m37.878684291s) [noise=true , address=45.91.116.9:4263 , piece_id=NS4BJHEUPAKAY2DBCP2CYH5BIADR6WGS3YBOADC7RSQWHNFD5JMA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.418720061 (360h3m37.880812544s) [noise=true , address=77.90.31.21:4879 , piece_id=NQQACZC75N2OVZEYYCSGOJRSDNN7I4MM6FK2S244OOTC3FNB6APQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.502015021 (360h3m37.964107504s) [noise=true , address=45.91.116.12:4226 , piece_id=A57OSOCQPWEBVNYNUPRIR3CHOSBHL2BMO4TM3QRM37D4MIUUQYJA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.503046987 (360h3m37.96513981s) [noise=true , address=45.91.116.14:4191 , piece_id=6GSU5AITXXNDAPZFGXXSEG4ELSD6YWIT5NJDOH2UOG4W2Z6WKA5A, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.526911169 (360h3m37.989003652s) [noise=true , address=77.90.31.12:4592 , piece_id=UJWSOJP73TKIWPZL2PTUVRLYLSG75X7WS5GFTTKP63B5MXNIA5MQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.526948269 (360h3m37.989040742s) [seg_pos={0 0}, upload=316776]: Segment reached optimal threshold of 13 pieces. Oct 8 13:34:52.527109416 (360h3m37.98920191s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 8 13:34:52.527120828 (360h3m37.989213311s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527176323 (360h3m37.989268806s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection Oct 8 13:34:52.527204477 (360h3m37.98929696s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Oct 8 13:34:52.527214617 (360h3m37.98930711s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527251387 (360h3m37.98934388s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527271655 (360h3m37.989364138s) [seg_pos={0 0}, upload=316776]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:34:52.527359282 (360h3m37.989451765s) [upload=316776]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:34:52.527366355 (360h3m37.989458848s) [upload=316776]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:34:52.880044686 (360h3m38.342137159s) [upload=316776]: Done uploading segments... Oct 8 13:34:52.880074503 (360h3m38.342166976s) [upload=316776]: Upload finished. err: <nil> Duration: 25.5689531s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....XX..XX...X.................X...................X...X...] duration: 8.425324508s FAIL (2025-10-08 13:38:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 13:34:49.838828743 (360h3m35.300921216s) [upload=316776]: Starting upload Oct 8 13:34:49.838915268 (360h3m35.301007741s) [upload=316776]: Done starting upload Oct 8 13:34:49.839094489 (360h3m35.301186962s) [upload=316776]: Uploading segments... Oct 8 13:34:49.839156237 (360h3m35.30124872s) [upload=316776]: Got next segment. Inline: false Oct 8 13:34:49.839191444 (360h3m35.301283917s) [upload=316776]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:34:49.839196824 (360h3m35.301289297s) [upload=316776]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:34:50.135260881 (360h3m35.597353354s) [seg_pos={0 0}, upload=316776]: Begin upload segment... Oct 8 13:34:50.135320996 (360h3m35.597413469s) [seg_pos={0 0}, upload=316776]: Done begin upload segment. Oct 8 13:34:50.135358337 (360h3m35.59745081s) [noise=true , address=77.90.31.21:4879 , piece_id=NQQACZC75N2OVZEYYCSGOJRSDNN7I4MM6FK2S244OOTC3FNB6APQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135365360 (360h3m35.597457843s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135441725 (360h3m35.597534198s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5EEIL6ATKQFJOYC4EFFGS2XTDN5OI6XXFEHRX2S222LEOHCQ4UQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135879640 (360h3m35.597972123s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.136377998 (360h3m35.598470481s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QY4YQ675MWT2RZPWNZ5AHX3KD22YBDTD2OTDQPT2EEUNTOPOTQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.136750097 (360h3m35.59884257s) [noise=true , address=45.91.116.14:4191 , piece_id=6GSU5AITXXNDAPZFGXXSEG4ELSD6YWIT5NJDOH2UOG4W2Z6WKA5A, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.137145740 (360h3m35.599238223s) [noise=true , address=145.79.227.13:4801 , piece_id=YLRCBTEMK7OL5B2F6L7OJGE6547YXBWPYSVHQRNLKTWC4G2TAN2A, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.138479160 (360h3m35.600571633s) [noise=true , address=77.90.31.11:4554 , piece_id=ZTJHP7NP4JSUTEFVYH6BETGYXZTVBSL3UYZL6ACD6U44KHQLTUNA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.139189071 (360h3m35.601281544s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.139862864 (360h3m35.601955347s) [noise=true , address=45.91.116.12:4226 , piece_id=A57OSOCQPWEBVNYNUPRIR3CHOSBHL2BMO4TM3QRM37D4MIUUQYJA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.140477925 (360h3m35.602570408s) [noise=true , address=45.91.116.9:4263 , piece_id=NS4BJHEUPAKAY2DBCP2CYH5BIADR6WGS3YBOADC7RSQWHNFD5JMA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.141060986 (360h3m35.603153469s) [noise=true , address=77.90.31.12:4592 , piece_id=UJWSOJP73TKIWPZL2PTUVRLYLSG75X7WS5GFTTKP63B5MXNIA5MQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.141495383 (360h3m35.603587866s) [noise=true , address=45.91.116.11:4119 , piece_id=4KGVJZZ5OSH6JUANPHHLRGYTAQECJAWP3HHTALP27XMSL57BB5EQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.142988015 (360h3m35.605080488s) [noise=true , address=45.91.116.6:4974 , piece_id=YUN4QEGIC4XY2AVGLJHEOJQG27JKCERNO6XTVEHGXT3K25TG24EQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.144130370 (360h3m35.606222853s) [noise=true , address=145.79.227.3:4151 , piece_id=XMJBO3O2BQHP5CDRGIKCMFCSQUXWP2FWEF7A3YKXTHSP4NR6Y6QQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.145590350 (360h3m35.607682834s) [noise=true , address=77.90.31.17:4770 , piece_id=4YUB24NXX4TCQSXVGHOWPQBDSX3PGZJJ2G2WIIVYOJVYISDVUKHQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:51.846370751 (360h3m37.308463224s) [upload=316776]: Next returned nil segment Oct 8 13:34:51.846393184 (360h3m37.308485657s) [upload=316776]: Waiting for error group managing segments... Oct 8 13:34:52.125182896 (360h3m37.587275369s) [noise=true , address=145.79.227.13:4801 , piece_id=YLRCBTEMK7OL5B2F6L7OJGE6547YXBWPYSVHQRNLKTWC4G2TAN2A, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.142347141 (360h3m37.604439624s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5EEIL6ATKQFJOYC4EFFGS2XTDN5OI6XXFEHRX2S222LEOHCQ4UQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.145727207 (360h3m37.60781969s) [noise=true , address=145.79.227.3:4151 , piece_id=XMJBO3O2BQHP5CDRGIKCMFCSQUXWP2FWEF7A3YKXTHSP4NR6Y6QQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.321404612 (360h3m37.783497085s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QY4YQ675MWT2RZPWNZ5AHX3KD22YBDTD2OTDQPT2EEUNTOPOTQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.392060135 (360h3m37.854152618s) [noise=true , address=45.91.116.11:4119 , piece_id=4KGVJZZ5OSH6JUANPHHLRGYTAQECJAWP3HHTALP27XMSL57BB5EQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.401936852 (360h3m37.864029335s) [noise=true , address=45.91.116.6:4974 , piece_id=YUN4QEGIC4XY2AVGLJHEOJQG27JKCERNO6XTVEHGXT3K25TG24EQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.408003444 (360h3m37.870095927s) [noise=true , address=77.90.31.11:4554 , piece_id=ZTJHP7NP4JSUTEFVYH6BETGYXZTVBSL3UYZL6ACD6U44KHQLTUNA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.411284171 (360h3m37.873376655s) [noise=true , address=77.90.31.17:4770 , piece_id=4YUB24NXX4TCQSXVGHOWPQBDSX3PGZJJ2G2WIIVYOJVYISDVUKHQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.416591808 (360h3m37.878684291s) [noise=true , address=45.91.116.9:4263 , piece_id=NS4BJHEUPAKAY2DBCP2CYH5BIADR6WGS3YBOADC7RSQWHNFD5JMA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.418720061 (360h3m37.880812544s) [noise=true , address=77.90.31.21:4879 , piece_id=NQQACZC75N2OVZEYYCSGOJRSDNN7I4MM6FK2S244OOTC3FNB6APQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.502015021 (360h3m37.964107504s) [noise=true , address=45.91.116.12:4226 , piece_id=A57OSOCQPWEBVNYNUPRIR3CHOSBHL2BMO4TM3QRM37D4MIUUQYJA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.503046987 (360h3m37.96513981s) [noise=true , address=45.91.116.14:4191 , piece_id=6GSU5AITXXNDAPZFGXXSEG4ELSD6YWIT5NJDOH2UOG4W2Z6WKA5A, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.526911169 (360h3m37.989003652s) [noise=true , address=77.90.31.12:4592 , piece_id=UJWSOJP73TKIWPZL2PTUVRLYLSG75X7WS5GFTTKP63B5MXNIA5MQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.526948269 (360h3m37.989040742s) [seg_pos={0 0}, upload=316776]: Segment reached optimal threshold of 13 pieces. Oct 8 13:34:52.527109416 (360h3m37.98920191s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 8 13:34:52.527120828 (360h3m37.989213311s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527176323 (360h3m37.989268806s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection Oct 8 13:34:52.527204477 (360h3m37.98929696s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Oct 8 13:34:52.527214617 (360h3m37.98930711s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527251387 (360h3m37.98934388s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527271655 (360h3m37.989364138s) [seg_pos={0 0}, upload=316776]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:34:52.527359282 (360h3m37.989451765s) [upload=316776]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:34:52.527366355 (360h3m37.989458848s) [upload=316776]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:34:52.880044686 (360h3m38.342137159s) [upload=316776]: Done uploading segments... Oct 8 13:34:52.880074503 (360h3m38.342166976s) [upload=316776]: Upload finished. err: <nil> Duration: 8.425360244s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......XX..XX...X.................X...................X...X..] duration: 6.562647156s FAIL (2025-10-08 13:37:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 13:34:49.838828743 (360h3m35.300921216s) [upload=316776]: Starting upload Oct 8 13:34:49.838915268 (360h3m35.301007741s) [upload=316776]: Done starting upload Oct 8 13:34:49.839094489 (360h3m35.301186962s) [upload=316776]: Uploading segments... Oct 8 13:34:49.839156237 (360h3m35.30124872s) [upload=316776]: Got next segment. Inline: false Oct 8 13:34:49.839191444 (360h3m35.301283917s) [upload=316776]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:34:49.839196824 (360h3m35.301289297s) [upload=316776]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:34:50.135260881 (360h3m35.597353354s) [seg_pos={0 0}, upload=316776]: Begin upload segment... Oct 8 13:34:50.135320996 (360h3m35.597413469s) [seg_pos={0 0}, upload=316776]: Done begin upload segment. Oct 8 13:34:50.135358337 (360h3m35.59745081s) [noise=true , address=77.90.31.21:4879 , piece_id=NQQACZC75N2OVZEYYCSGOJRSDNN7I4MM6FK2S244OOTC3FNB6APQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135365360 (360h3m35.597457843s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135441725 (360h3m35.597534198s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5EEIL6ATKQFJOYC4EFFGS2XTDN5OI6XXFEHRX2S222LEOHCQ4UQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135879640 (360h3m35.597972123s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.136377998 (360h3m35.598470481s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QY4YQ675MWT2RZPWNZ5AHX3KD22YBDTD2OTDQPT2EEUNTOPOTQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.136750097 (360h3m35.59884257s) [noise=true , address=45.91.116.14:4191 , piece_id=6GSU5AITXXNDAPZFGXXSEG4ELSD6YWIT5NJDOH2UOG4W2Z6WKA5A, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.137145740 (360h3m35.599238223s) [noise=true , address=145.79.227.13:4801 , piece_id=YLRCBTEMK7OL5B2F6L7OJGE6547YXBWPYSVHQRNLKTWC4G2TAN2A, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.138479160 (360h3m35.600571633s) [noise=true , address=77.90.31.11:4554 , piece_id=ZTJHP7NP4JSUTEFVYH6BETGYXZTVBSL3UYZL6ACD6U44KHQLTUNA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.139189071 (360h3m35.601281544s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.139862864 (360h3m35.601955347s) [noise=true , address=45.91.116.12:4226 , piece_id=A57OSOCQPWEBVNYNUPRIR3CHOSBHL2BMO4TM3QRM37D4MIUUQYJA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.140477925 (360h3m35.602570408s) [noise=true , address=45.91.116.9:4263 , piece_id=NS4BJHEUPAKAY2DBCP2CYH5BIADR6WGS3YBOADC7RSQWHNFD5JMA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.141060986 (360h3m35.603153469s) [noise=true , address=77.90.31.12:4592 , piece_id=UJWSOJP73TKIWPZL2PTUVRLYLSG75X7WS5GFTTKP63B5MXNIA5MQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.141495383 (360h3m35.603587866s) [noise=true , address=45.91.116.11:4119 , piece_id=4KGVJZZ5OSH6JUANPHHLRGYTAQECJAWP3HHTALP27XMSL57BB5EQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.142988015 (360h3m35.605080488s) [noise=true , address=45.91.116.6:4974 , piece_id=YUN4QEGIC4XY2AVGLJHEOJQG27JKCERNO6XTVEHGXT3K25TG24EQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.144130370 (360h3m35.606222853s) [noise=true , address=145.79.227.3:4151 , piece_id=XMJBO3O2BQHP5CDRGIKCMFCSQUXWP2FWEF7A3YKXTHSP4NR6Y6QQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.145590350 (360h3m35.607682834s) [noise=true , address=77.90.31.17:4770 , piece_id=4YUB24NXX4TCQSXVGHOWPQBDSX3PGZJJ2G2WIIVYOJVYISDVUKHQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:51.846370751 (360h3m37.308463224s) [upload=316776]: Next returned nil segment Oct 8 13:34:51.846393184 (360h3m37.308485657s) [upload=316776]: Waiting for error group managing segments... Oct 8 13:34:52.125182896 (360h3m37.587275369s) [noise=true , address=145.79.227.13:4801 , piece_id=YLRCBTEMK7OL5B2F6L7OJGE6547YXBWPYSVHQRNLKTWC4G2TAN2A, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.142347141 (360h3m37.604439624s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5EEIL6ATKQFJOYC4EFFGS2XTDN5OI6XXFEHRX2S222LEOHCQ4UQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.145727207 (360h3m37.60781969s) [noise=true , address=145.79.227.3:4151 , piece_id=XMJBO3O2BQHP5CDRGIKCMFCSQUXWP2FWEF7A3YKXTHSP4NR6Y6QQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.321404612 (360h3m37.783497085s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QY4YQ675MWT2RZPWNZ5AHX3KD22YBDTD2OTDQPT2EEUNTOPOTQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.392060135 (360h3m37.854152618s) [noise=true , address=45.91.116.11:4119 , piece_id=4KGVJZZ5OSH6JUANPHHLRGYTAQECJAWP3HHTALP27XMSL57BB5EQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.401936852 (360h3m37.864029335s) [noise=true , address=45.91.116.6:4974 , piece_id=YUN4QEGIC4XY2AVGLJHEOJQG27JKCERNO6XTVEHGXT3K25TG24EQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.408003444 (360h3m37.870095927s) [noise=true , address=77.90.31.11:4554 , piece_id=ZTJHP7NP4JSUTEFVYH6BETGYXZTVBSL3UYZL6ACD6U44KHQLTUNA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.411284171 (360h3m37.873376655s) [noise=true , address=77.90.31.17:4770 , piece_id=4YUB24NXX4TCQSXVGHOWPQBDSX3PGZJJ2G2WIIVYOJVYISDVUKHQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.416591808 (360h3m37.878684291s) [noise=true , address=45.91.116.9:4263 , piece_id=NS4BJHEUPAKAY2DBCP2CYH5BIADR6WGS3YBOADC7RSQWHNFD5JMA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.418720061 (360h3m37.880812544s) [noise=true , address=77.90.31.21:4879 , piece_id=NQQACZC75N2OVZEYYCSGOJRSDNN7I4MM6FK2S244OOTC3FNB6APQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.502015021 (360h3m37.964107504s) [noise=true , address=45.91.116.12:4226 , piece_id=A57OSOCQPWEBVNYNUPRIR3CHOSBHL2BMO4TM3QRM37D4MIUUQYJA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.503046987 (360h3m37.96513981s) [noise=true , address=45.91.116.14:4191 , piece_id=6GSU5AITXXNDAPZFGXXSEG4ELSD6YWIT5NJDOH2UOG4W2Z6WKA5A, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.526911169 (360h3m37.989003652s) [noise=true , address=77.90.31.12:4592 , piece_id=UJWSOJP73TKIWPZL2PTUVRLYLSG75X7WS5GFTTKP63B5MXNIA5MQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.526948269 (360h3m37.989040742s) [seg_pos={0 0}, upload=316776]: Segment reached optimal threshold of 13 pieces. Oct 8 13:34:52.527109416 (360h3m37.98920191s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 8 13:34:52.527120828 (360h3m37.989213311s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527176323 (360h3m37.989268806s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection Oct 8 13:34:52.527204477 (360h3m37.98929696s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Oct 8 13:34:52.527214617 (360h3m37.98930711s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527251387 (360h3m37.98934388s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527271655 (360h3m37.989364138s) [seg_pos={0 0}, upload=316776]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:34:52.527359282 (360h3m37.989451765s) [upload=316776]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:34:52.527366355 (360h3m37.989458848s) [upload=316776]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:34:52.880044686 (360h3m38.342137159s) [upload=316776]: Done uploading segments... Oct 8 13:34:52.880074503 (360h3m38.342166976s) [upload=316776]: Upload finished. err: <nil> Duration: 6.562676281s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......XX..XX...X.................X...................X...X.] duration: 19.930584741s FAIL (2025-10-08 13:36:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 13:34:49.838828743 (360h3m35.300921216s) [upload=316776]: Starting upload Oct 8 13:34:49.838915268 (360h3m35.301007741s) [upload=316776]: Done starting upload Oct 8 13:34:49.839094489 (360h3m35.301186962s) [upload=316776]: Uploading segments... Oct 8 13:34:49.839156237 (360h3m35.30124872s) [upload=316776]: Got next segment. Inline: false Oct 8 13:34:49.839191444 (360h3m35.301283917s) [upload=316776]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:34:49.839196824 (360h3m35.301289297s) [upload=316776]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:34:50.135260881 (360h3m35.597353354s) [seg_pos={0 0}, upload=316776]: Begin upload segment... Oct 8 13:34:50.135320996 (360h3m35.597413469s) [seg_pos={0 0}, upload=316776]: Done begin upload segment. Oct 8 13:34:50.135358337 (360h3m35.59745081s) [noise=true , address=77.90.31.21:4879 , piece_id=NQQACZC75N2OVZEYYCSGOJRSDNN7I4MM6FK2S244OOTC3FNB6APQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135365360 (360h3m35.597457843s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135441725 (360h3m35.597534198s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5EEIL6ATKQFJOYC4EFFGS2XTDN5OI6XXFEHRX2S222LEOHCQ4UQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135879640 (360h3m35.597972123s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.136377998 (360h3m35.598470481s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QY4YQ675MWT2RZPWNZ5AHX3KD22YBDTD2OTDQPT2EEUNTOPOTQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.136750097 (360h3m35.59884257s) [noise=true , address=45.91.116.14:4191 , piece_id=6GSU5AITXXNDAPZFGXXSEG4ELSD6YWIT5NJDOH2UOG4W2Z6WKA5A, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.137145740 (360h3m35.599238223s) [noise=true , address=145.79.227.13:4801 , piece_id=YLRCBTEMK7OL5B2F6L7OJGE6547YXBWPYSVHQRNLKTWC4G2TAN2A, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.138479160 (360h3m35.600571633s) [noise=true , address=77.90.31.11:4554 , piece_id=ZTJHP7NP4JSUTEFVYH6BETGYXZTVBSL3UYZL6ACD6U44KHQLTUNA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.139189071 (360h3m35.601281544s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.139862864 (360h3m35.601955347s) [noise=true , address=45.91.116.12:4226 , piece_id=A57OSOCQPWEBVNYNUPRIR3CHOSBHL2BMO4TM3QRM37D4MIUUQYJA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.140477925 (360h3m35.602570408s) [noise=true , address=45.91.116.9:4263 , piece_id=NS4BJHEUPAKAY2DBCP2CYH5BIADR6WGS3YBOADC7RSQWHNFD5JMA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.141060986 (360h3m35.603153469s) [noise=true , address=77.90.31.12:4592 , piece_id=UJWSOJP73TKIWPZL2PTUVRLYLSG75X7WS5GFTTKP63B5MXNIA5MQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.141495383 (360h3m35.603587866s) [noise=true , address=45.91.116.11:4119 , piece_id=4KGVJZZ5OSH6JUANPHHLRGYTAQECJAWP3HHTALP27XMSL57BB5EQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.142988015 (360h3m35.605080488s) [noise=true , address=45.91.116.6:4974 , piece_id=YUN4QEGIC4XY2AVGLJHEOJQG27JKCERNO6XTVEHGXT3K25TG24EQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.144130370 (360h3m35.606222853s) [noise=true , address=145.79.227.3:4151 , piece_id=XMJBO3O2BQHP5CDRGIKCMFCSQUXWP2FWEF7A3YKXTHSP4NR6Y6QQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.145590350 (360h3m35.607682834s) [noise=true , address=77.90.31.17:4770 , piece_id=4YUB24NXX4TCQSXVGHOWPQBDSX3PGZJJ2G2WIIVYOJVYISDVUKHQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:51.846370751 (360h3m37.308463224s) [upload=316776]: Next returned nil segment Oct 8 13:34:51.846393184 (360h3m37.308485657s) [upload=316776]: Waiting for error group managing segments... Oct 8 13:34:52.125182896 (360h3m37.587275369s) [noise=true , address=145.79.227.13:4801 , piece_id=YLRCBTEMK7OL5B2F6L7OJGE6547YXBWPYSVHQRNLKTWC4G2TAN2A, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.142347141 (360h3m37.604439624s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5EEIL6ATKQFJOYC4EFFGS2XTDN5OI6XXFEHRX2S222LEOHCQ4UQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.145727207 (360h3m37.60781969s) [noise=true , address=145.79.227.3:4151 , piece_id=XMJBO3O2BQHP5CDRGIKCMFCSQUXWP2FWEF7A3YKXTHSP4NR6Y6QQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.321404612 (360h3m37.783497085s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QY4YQ675MWT2RZPWNZ5AHX3KD22YBDTD2OTDQPT2EEUNTOPOTQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.392060135 (360h3m37.854152618s) [noise=true , address=45.91.116.11:4119 , piece_id=4KGVJZZ5OSH6JUANPHHLRGYTAQECJAWP3HHTALP27XMSL57BB5EQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.401936852 (360h3m37.864029335s) [noise=true , address=45.91.116.6:4974 , piece_id=YUN4QEGIC4XY2AVGLJHEOJQG27JKCERNO6XTVEHGXT3K25TG24EQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.408003444 (360h3m37.870095927s) [noise=true , address=77.90.31.11:4554 , piece_id=ZTJHP7NP4JSUTEFVYH6BETGYXZTVBSL3UYZL6ACD6U44KHQLTUNA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.411284171 (360h3m37.873376655s) [noise=true , address=77.90.31.17:4770 , piece_id=4YUB24NXX4TCQSXVGHOWPQBDSX3PGZJJ2G2WIIVYOJVYISDVUKHQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.416591808 (360h3m37.878684291s) [noise=true , address=45.91.116.9:4263 , piece_id=NS4BJHEUPAKAY2DBCP2CYH5BIADR6WGS3YBOADC7RSQWHNFD5JMA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.418720061 (360h3m37.880812544s) [noise=true , address=77.90.31.21:4879 , piece_id=NQQACZC75N2OVZEYYCSGOJRSDNN7I4MM6FK2S244OOTC3FNB6APQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.502015021 (360h3m37.964107504s) [noise=true , address=45.91.116.12:4226 , piece_id=A57OSOCQPWEBVNYNUPRIR3CHOSBHL2BMO4TM3QRM37D4MIUUQYJA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.503046987 (360h3m37.96513981s) [noise=true , address=45.91.116.14:4191 , piece_id=6GSU5AITXXNDAPZFGXXSEG4ELSD6YWIT5NJDOH2UOG4W2Z6WKA5A, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.526911169 (360h3m37.989003652s) [noise=true , address=77.90.31.12:4592 , piece_id=UJWSOJP73TKIWPZL2PTUVRLYLSG75X7WS5GFTTKP63B5MXNIA5MQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.526948269 (360h3m37.989040742s) [seg_pos={0 0}, upload=316776]: Segment reached optimal threshold of 13 pieces. Oct 8 13:34:52.527109416 (360h3m37.98920191s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 8 13:34:52.527120828 (360h3m37.989213311s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527176323 (360h3m37.989268806s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection Oct 8 13:34:52.527204477 (360h3m37.98929696s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Oct 8 13:34:52.527214617 (360h3m37.98930711s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527251387 (360h3m37.98934388s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527271655 (360h3m37.989364138s) [seg_pos={0 0}, upload=316776]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:34:52.527359282 (360h3m37.989451765s) [upload=316776]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:34:52.527366355 (360h3m37.989458848s) [upload=316776]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:34:52.880044686 (360h3m38.342137159s) [upload=316776]: Done uploading segments... Oct 8 13:34:52.880074503 (360h3m38.342166976s) [upload=316776]: Upload finished. err: <nil> Duration: 19.93062174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........XX..XX...X.................X...................X...X] duration: 30.002912046s FAIL (2025-10-08 13:35:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 8 13:34:49.838828743 (360h3m35.300921216s) [upload=316776]: Starting upload Oct 8 13:34:49.838915268 (360h3m35.301007741s) [upload=316776]: Done starting upload Oct 8 13:34:49.839094489 (360h3m35.301186962s) [upload=316776]: Uploading segments... Oct 8 13:34:49.839156237 (360h3m35.30124872s) [upload=316776]: Got next segment. Inline: false Oct 8 13:34:49.839191444 (360h3m35.301283917s) [upload=316776]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:34:49.839196824 (360h3m35.301289297s) [upload=316776]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:34:50.135260881 (360h3m35.597353354s) [seg_pos={0 0}, upload=316776]: Begin upload segment... Oct 8 13:34:50.135320996 (360h3m35.597413469s) [seg_pos={0 0}, upload=316776]: Done begin upload segment. Oct 8 13:34:50.135358337 (360h3m35.59745081s) [noise=true , address=77.90.31.21:4879 , piece_id=NQQACZC75N2OVZEYYCSGOJRSDNN7I4MM6FK2S244OOTC3FNB6APQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135365360 (360h3m35.597457843s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135441725 (360h3m35.597534198s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5EEIL6ATKQFJOYC4EFFGS2XTDN5OI6XXFEHRX2S222LEOHCQ4UQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.135879640 (360h3m35.597972123s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.136377998 (360h3m35.598470481s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QY4YQ675MWT2RZPWNZ5AHX3KD22YBDTD2OTDQPT2EEUNTOPOTQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.136750097 (360h3m35.59884257s) [noise=true , address=45.91.116.14:4191 , piece_id=6GSU5AITXXNDAPZFGXXSEG4ELSD6YWIT5NJDOH2UOG4W2Z6WKA5A, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.137145740 (360h3m35.599238223s) [noise=true , address=145.79.227.13:4801 , piece_id=YLRCBTEMK7OL5B2F6L7OJGE6547YXBWPYSVHQRNLKTWC4G2TAN2A, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.138479160 (360h3m35.600571633s) [noise=true , address=77.90.31.11:4554 , piece_id=ZTJHP7NP4JSUTEFVYH6BETGYXZTVBSL3UYZL6ACD6U44KHQLTUNA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.139189071 (360h3m35.601281544s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.139862864 (360h3m35.601955347s) [noise=true , address=45.91.116.12:4226 , piece_id=A57OSOCQPWEBVNYNUPRIR3CHOSBHL2BMO4TM3QRM37D4MIUUQYJA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.140477925 (360h3m35.602570408s) [noise=true , address=45.91.116.9:4263 , piece_id=NS4BJHEUPAKAY2DBCP2CYH5BIADR6WGS3YBOADC7RSQWHNFD5JMA, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.141060986 (360h3m35.603153469s) [noise=true , address=77.90.31.12:4592 , piece_id=UJWSOJP73TKIWPZL2PTUVRLYLSG75X7WS5GFTTKP63B5MXNIA5MQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.141495383 (360h3m35.603587866s) [noise=true , address=45.91.116.11:4119 , piece_id=4KGVJZZ5OSH6JUANPHHLRGYTAQECJAWP3HHTALP27XMSL57BB5EQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.142988015 (360h3m35.605080488s) [noise=true , address=45.91.116.6:4974 , piece_id=YUN4QEGIC4XY2AVGLJHEOJQG27JKCERNO6XTVEHGXT3K25TG24EQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.144130370 (360h3m35.606222853s) [noise=true , address=145.79.227.3:4151 , piece_id=XMJBO3O2BQHP5CDRGIKCMFCSQUXWP2FWEF7A3YKXTHSP4NR6Y6QQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:50.145590350 (360h3m35.607682834s) [noise=true , address=77.90.31.17:4770 , piece_id=4YUB24NXX4TCQSXVGHOWPQBDSX3PGZJJ2G2WIIVYOJVYISDVUKHQ, seg_pos={0 0}, upload=316776]: Uploading piece... Oct 8 13:34:51.846370751 (360h3m37.308463224s) [upload=316776]: Next returned nil segment Oct 8 13:34:51.846393184 (360h3m37.308485657s) [upload=316776]: Waiting for error group managing segments... Oct 8 13:34:52.125182896 (360h3m37.587275369s) [noise=true , address=145.79.227.13:4801 , piece_id=YLRCBTEMK7OL5B2F6L7OJGE6547YXBWPYSVHQRNLKTWC4G2TAN2A, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.142347141 (360h3m37.604439624s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5EEIL6ATKQFJOYC4EFFGS2XTDN5OI6XXFEHRX2S222LEOHCQ4UQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.145727207 (360h3m37.60781969s) [noise=true , address=145.79.227.3:4151 , piece_id=XMJBO3O2BQHP5CDRGIKCMFCSQUXWP2FWEF7A3YKXTHSP4NR6Y6QQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.321404612 (360h3m37.783497085s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QY4YQ675MWT2RZPWNZ5AHX3KD22YBDTD2OTDQPT2EEUNTOPOTQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.392060135 (360h3m37.854152618s) [noise=true , address=45.91.116.11:4119 , piece_id=4KGVJZZ5OSH6JUANPHHLRGYTAQECJAWP3HHTALP27XMSL57BB5EQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.401936852 (360h3m37.864029335s) [noise=true , address=45.91.116.6:4974 , piece_id=YUN4QEGIC4XY2AVGLJHEOJQG27JKCERNO6XTVEHGXT3K25TG24EQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.408003444 (360h3m37.870095927s) [noise=true , address=77.90.31.11:4554 , piece_id=ZTJHP7NP4JSUTEFVYH6BETGYXZTVBSL3UYZL6ACD6U44KHQLTUNA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.411284171 (360h3m37.873376655s) [noise=true , address=77.90.31.17:4770 , piece_id=4YUB24NXX4TCQSXVGHOWPQBDSX3PGZJJ2G2WIIVYOJVYISDVUKHQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.416591808 (360h3m37.878684291s) [noise=true , address=45.91.116.9:4263 , piece_id=NS4BJHEUPAKAY2DBCP2CYH5BIADR6WGS3YBOADC7RSQWHNFD5JMA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.418720061 (360h3m37.880812544s) [noise=true , address=77.90.31.21:4879 , piece_id=NQQACZC75N2OVZEYYCSGOJRSDNN7I4MM6FK2S244OOTC3FNB6APQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.502015021 (360h3m37.964107504s) [noise=true , address=45.91.116.12:4226 , piece_id=A57OSOCQPWEBVNYNUPRIR3CHOSBHL2BMO4TM3QRM37D4MIUUQYJA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.503046987 (360h3m37.96513981s) [noise=true , address=45.91.116.14:4191 , piece_id=6GSU5AITXXNDAPZFGXXSEG4ELSD6YWIT5NJDOH2UOG4W2Z6WKA5A, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.526911169 (360h3m37.989003652s) [noise=true , address=77.90.31.12:4592 , piece_id=UJWSOJP73TKIWPZL2PTUVRLYLSG75X7WS5GFTTKP63B5MXNIA5MQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: <nil> Oct 8 13:34:52.526948269 (360h3m37.989040742s) [seg_pos={0 0}, upload=316776]: Segment reached optimal threshold of 13 pieces. Oct 8 13:34:52.527109416 (360h3m37.98920191s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 8 13:34:52.527120828 (360h3m37.989213311s) [noise=true , address=77.90.31.16:4733 , piece_id=ANYGV2GPG2K4PV6MERWFYK5M4DMTJRWGEOCUEERGA3KFBDFMWUHA, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527176323 (360h3m37.989268806s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37974->45.91.116.13:4153: use of closed network connection Oct 8 13:34:52.527204477 (360h3m37.98929696s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Oct 8 13:34:52.527214617 (360h3m37.98930711s) [noise=true , address=77.90.31.19:4842 , piece_id=SYKJNJSIXGRHEZVPPNQYSU4INEDHVEDG4QVWJZ5ISARCDABSOZKQ, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527251387 (360h3m37.98934388s) [noise=true , address=45.91.116.13:4153 , piece_id=6YCHK5QCZDG724HJVZGRWYGUFBXDKA6PNSWSB4CW5QQ3BT5FXHIA, seg_pos={0 0}, upload=316776]: Long tail cancellation due to optimal threshold Oct 8 13:34:52.527271655 (360h3m37.989364138s) [seg_pos={0 0}, upload=316776]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:34:52.527359282 (360h3m37.989451765s) [upload=316776]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:34:52.527366355 (360h3m37.989458848s) [upload=316776]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:34:52.880044686 (360h3m38.342137159s) [upload=316776]: Done uploading segments... Oct 8 13:34:52.880074503 (360h3m38.342166976s) [upload=316776]: Upload finished. err: <nil> Duration: 30.002942705s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X........XX..XX...X.................X...................X...] duration: 6.629213532s FAIL (2025-10-08 13:34:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 2BIYVGPYKNOCG4S3E4QTUWQUJVX64TEFXQV2NI2CTRMLLZX4KJNA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: O2TLURC6WLQQZIZBBNCEN2TRK42ZXQN2KGWGXC7QV34GFLCJMA5Q) context deadline exceeded ======LOGS====== Oct 8 13:31:42.509127899 (360h0m27.971220372s) [upload=316734]: Starting upload Oct 8 13:31:42.509230875 (360h0m27.971323358s) [upload=316734]: Done starting upload Oct 8 13:31:42.509359320 (360h0m27.971451813s) [upload=316734]: Uploading segments... Oct 8 13:31:42.509483116 (360h0m27.971575599s) [upload=316734]: Got next segment. Inline: false Oct 8 13:31:42.509529254 (360h0m27.971621737s) [upload=316734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:31:42.509534975 (360h0m27.971627448s) [upload=316734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:31:42.757350406 (360h0m28.219442889s) [seg_pos={0 0}, upload=316734]: Begin upload segment... Oct 8 13:31:42.757395101 (360h0m28.219487584s) [seg_pos={0 0}, upload=316734]: Done begin upload segment. Oct 8 13:31:42.757436630 (360h0m28.219529113s) [noise=true , address=45.91.116.7:4009 , piece_id=O2TLURC6WLQQZIZBBNCEN2TRK42ZXQN2KGWGXC7QV34GFLCJMA5Q, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.758244649 (360h0m28.220337132s) [noise=true , address=145.79.227.4:4187 , piece_id=JHWEFNAVWBOSHXR6TEJJRQYWOCRPPMF4275PADBLE7DQ5GCEQ3RQ, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.758610716 (360h0m28.220703199s) [noise=true , address=45.91.116.5:4298 , piece_id=7E24KJ76NH6DE73HZRMIX2DENSTP2M6JFZEUSIHUWJNZN36TANTQ, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.758956725 (360h0m28.221049198s) [noise=true , address=145.79.227.7:4295 , piece_id=SAF3CDB66F2REAWVSMRUNOK7B22CRHHANICMZT2RYTF6NVPOPOIA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.759373288 (360h0m28.221465771s) [noise=true , address=145.79.227.11:4729 , piece_id=32VVCWGARNJZBETVKPQ4AYDPT7EZ4JAKNYSD7KBISQEJTKTBDRKA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.759845327 (360h0m28.22193781s) [noise=true , address=77.90.31.16:4733 , piece_id=QEEUYHGBOH7SVXDSZL3GEN6HTZQDZJEDYVPJTXM3HR7SERMVRV5A, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.760140038 (360h0m28.222232511s) [noise=true , address=145.79.227.5:4222 , piece_id=BNK2OZZ2I2BVGCW4Y2MUYXLEOCPTWJID37WR4ZIQ7RFXZGBPH6YQ, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.760506827 (360h0m28.22259931s) [noise=true , address=45.91.116.8:4048 , piece_id=2BIYVGPYKNOCG4S3E4QTUWQUJVX64TEFXQV2NI2CTRMLLZX4KJNA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.760848366 (360h0m28.222940849s) [noise=true , address=77.90.31.13:4626 , piece_id=6LKA3AE6QYGFCE6ZKBEXNEODMECL4LMP6ARE5OP7RCOZQF2KZGRA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.761135193 (360h0m28.223227666s) [noise=true , address=77.90.31.14:4662 , piece_id=QTLWLCAG7MGVSYHG6BH65GPBIBEVEYRU7J56UAY4TC763PRSJ3MA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.761410337 (360h0m28.22350282s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGB2JAC3CMNKBE24VL6R2YS5ZQXJ6PREJ4UHMM3L4CUTQCJ4J4BA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.761723053 (360h0m28.223815526s) [noise=true , address=145.79.227.19:4079 , piece_id=HC3FC7XPGUTY7QANZD4FPRDXO7YLQIFKDSTVUHLOYLX5WG5EVVDQ, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.762034274 (360h0m28.224126747s) [noise=true , address=77.90.31.12:4589 , piece_id=4CASAPHOLRH7YGEMUJB33Z435N7SBBDHM7WOXEF7VXWEXCFNFAVA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.762300682 (360h0m28.224393155s) [noise=true , address=145.79.227.17:4971 , piece_id=G2JMMSEDTDJLWLKEMXODXY7DM3OIOTNDTRELBN3CZ4ICGGWJI73Q, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.762587067 (360h0m28.22467954s) [noise=true , address=145.79.227.15:4875 , piece_id=VX6S33AND7I7P6XONK575ISXAHKICATJ7NYSTPDAQ64CQJRDHFRA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.762857192 (360h0m28.224949675s) [noise=true , address=77.90.31.17:4771 , piece_id=WO645Q2EACGUPHLYDUK7245WORG6WMCOL36ZAZAULLZHRFM4SENA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:44.466626765 (360h0m29.928719248s) [upload=316734]: Next returned nil segment Oct 8 13:31:44.466648777 (360h0m29.92874125s) [upload=316734]: Waiting for error group managing segments... Oct 8 13:31:44.744695795 (360h0m30.206788268s) [noise=true , address=145.79.227.7:4295 , piece_id=SAF3CDB66F2REAWVSMRUNOK7B22CRHHANICMZT2RYTF6NVPOPOIA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.746695834 (360h0m30.208788317s) [noise=true , address=145.79.227.19:4079 , piece_id=HC3FC7XPGUTY7QANZD4FPRDXO7YLQIFKDSTVUHLOYLX5WG5EVVDQ, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.747509312 (360h0m30.209601795s) [noise=true , address=145.79.227.15:4875 , piece_id=VX6S33AND7I7P6XONK575ISXAHKICATJ7NYSTPDAQ64CQJRDHFRA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.754168252 (360h0m30.216260725s) [noise=true , address=145.79.227.4:4187 , piece_id=JHWEFNAVWBOSHXR6TEJJRQYWOCRPPMF4275PADBLE7DQ5GCEQ3RQ, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.760404758 (360h0m30.222497231s) [noise=true , address=145.79.227.11:4729 , piece_id=32VVCWGARNJZBETVKPQ4AYDPT7EZ4JAKNYSD7KBISQEJTKTBDRKA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.767948112 (360h0m30.230040595s) [noise=true , address=145.79.227.17:4971 , piece_id=G2JMMSEDTDJLWLKEMXODXY7DM3OIOTNDTRELBN3CZ4ICGGWJI73Q, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.797366731 (360h0m30.259459204s) [noise=true , address=145.79.227.5:4222 , piece_id=BNK2OZZ2I2BVGCW4Y2MUYXLEOCPTWJID37WR4ZIQ7RFXZGBPH6YQ, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.973460900 (360h0m30.435553373s) [noise=true , address=45.91.116.7:4009 , piece_id=O2TLURC6WLQQZIZBBNCEN2TRK42ZXQN2KGWGXC7QV34GFLCJMA5Q, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.985872144 (360h0m30.447964627s) [noise=true , address=45.91.116.5:4298 , piece_id=7E24KJ76NH6DE73HZRMIX2DENSTP2M6JFZEUSIHUWJNZN36TANTQ, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.059911309 (360h0m30.522003792s) [noise=true , address=45.91.116.8:4048 , piece_id=2BIYVGPYKNOCG4S3E4QTUWQUJVX64TEFXQV2NI2CTRMLLZX4KJNA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.060621962 (360h0m30.522714445s) [noise=true , address=77.90.31.17:4771 , piece_id=WO645Q2EACGUPHLYDUK7245WORG6WMCOL36ZAZAULLZHRFM4SENA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.075465818 (360h0m30.537558291s) [noise=true , address=77.90.31.16:4733 , piece_id=QEEUYHGBOH7SVXDSZL3GEN6HTZQDZJEDYVPJTXM3HR7SERMVRV5A, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.105250084 (360h0m30.567342557s) [noise=true , address=77.90.31.13:4626 , piece_id=6LKA3AE6QYGFCE6ZKBEXNEODMECL4LMP6ARE5OP7RCOZQF2KZGRA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.105305460 (360h0m30.567397933s) [seg_pos={0 0}, upload=316734]: Segment reached optimal threshold of 13 pieces. Oct 8 13:31:45.105442179 (360h0m30.567534652s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGB2JAC3CMNKBE24VL6R2YS5ZQXJ6PREJ4UHMM3L4CUTQCJ4J4BA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Oct 8 13:31:45.105449103 (360h0m30.567541576s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGB2JAC3CMNKBE24VL6R2YS5ZQXJ6PREJ4UHMM3L4CUTQCJ4J4BA, seg_pos={0 0}, upload=316734]: Long tail cancellation due to optimal threshold Oct 8 13:31:45.105522753 (360h0m30.567615236s) [noise=true , address=77.90.31.12:4589 , piece_id=4CASAPHOLRH7YGEMUJB33Z435N7SBBDHM7WOXEF7VXWEXCFNFAVA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Oct 8 13:31:45.105528474 (360h0m30.567620957s) [noise=true , address=77.90.31.12:4589 , piece_id=4CASAPHOLRH7YGEMUJB33Z435N7SBBDHM7WOXEF7VXWEXCFNFAVA, seg_pos={0 0}, upload=316734]: Long tail cancellation due to optimal threshold Oct 8 13:31:45.105568289 (360h0m30.567660763s) [noise=true , address=77.90.31.14:4662 , piece_id=QTLWLCAG7MGVSYHG6BH65GPBIBEVEYRU7J56UAY4TC763PRSJ3MA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 13:31:45.105588929 (360h0m30.567681402s) [noise=true , address=77.90.31.14:4662 , piece_id=QTLWLCAG7MGVSYHG6BH65GPBIBEVEYRU7J56UAY4TC763PRSJ3MA, seg_pos={0 0}, upload=316734]: Long tail cancellation due to optimal threshold Oct 8 13:31:45.105621772 (360h0m30.567714255s) [seg_pos={0 0}, upload=316734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:31:45.105706332 (360h0m30.567798815s) [upload=316734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:31:45.105713526 (360h0m30.567805999s) [upload=316734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:31:45.548510072 (360h0m31.010602555s) [upload=316734]: Done uploading segments... Oct 8 13:31:45.548533807 (360h0m31.01062629s) [upload=316734]: Upload finished. err: <nil> Duration: 6.629245614s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X........XX..XX...X.................X...................X..] duration: 27.063597051s FAIL (2025-10-08 13:34:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 2BIYVGPYKNOCG4S3E4QTUWQUJVX64TEFXQV2NI2CTRMLLZX4KJNA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: O2TLURC6WLQQZIZBBNCEN2TRK42ZXQN2KGWGXC7QV34GFLCJMA5Q) context deadline exceeded ======LOGS====== Oct 8 13:31:42.509127899 (360h0m27.971220372s) [upload=316734]: Starting upload Oct 8 13:31:42.509230875 (360h0m27.971323358s) [upload=316734]: Done starting upload Oct 8 13:31:42.509359320 (360h0m27.971451813s) [upload=316734]: Uploading segments... Oct 8 13:31:42.509483116 (360h0m27.971575599s) [upload=316734]: Got next segment. Inline: false Oct 8 13:31:42.509529254 (360h0m27.971621737s) [upload=316734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:31:42.509534975 (360h0m27.971627448s) [upload=316734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:31:42.757350406 (360h0m28.219442889s) [seg_pos={0 0}, upload=316734]: Begin upload segment... Oct 8 13:31:42.757395101 (360h0m28.219487584s) [seg_pos={0 0}, upload=316734]: Done begin upload segment. Oct 8 13:31:42.757436630 (360h0m28.219529113s) [noise=true , address=45.91.116.7:4009 , piece_id=O2TLURC6WLQQZIZBBNCEN2TRK42ZXQN2KGWGXC7QV34GFLCJMA5Q, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.758244649 (360h0m28.220337132s) [noise=true , address=145.79.227.4:4187 , piece_id=JHWEFNAVWBOSHXR6TEJJRQYWOCRPPMF4275PADBLE7DQ5GCEQ3RQ, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.758610716 (360h0m28.220703199s) [noise=true , address=45.91.116.5:4298 , piece_id=7E24KJ76NH6DE73HZRMIX2DENSTP2M6JFZEUSIHUWJNZN36TANTQ, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.758956725 (360h0m28.221049198s) [noise=true , address=145.79.227.7:4295 , piece_id=SAF3CDB66F2REAWVSMRUNOK7B22CRHHANICMZT2RYTF6NVPOPOIA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.759373288 (360h0m28.221465771s) [noise=true , address=145.79.227.11:4729 , piece_id=32VVCWGARNJZBETVKPQ4AYDPT7EZ4JAKNYSD7KBISQEJTKTBDRKA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.759845327 (360h0m28.22193781s) [noise=true , address=77.90.31.16:4733 , piece_id=QEEUYHGBOH7SVXDSZL3GEN6HTZQDZJEDYVPJTXM3HR7SERMVRV5A, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.760140038 (360h0m28.222232511s) [noise=true , address=145.79.227.5:4222 , piece_id=BNK2OZZ2I2BVGCW4Y2MUYXLEOCPTWJID37WR4ZIQ7RFXZGBPH6YQ, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.760506827 (360h0m28.22259931s) [noise=true , address=45.91.116.8:4048 , piece_id=2BIYVGPYKNOCG4S3E4QTUWQUJVX64TEFXQV2NI2CTRMLLZX4KJNA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.760848366 (360h0m28.222940849s) [noise=true , address=77.90.31.13:4626 , piece_id=6LKA3AE6QYGFCE6ZKBEXNEODMECL4LMP6ARE5OP7RCOZQF2KZGRA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.761135193 (360h0m28.223227666s) [noise=true , address=77.90.31.14:4662 , piece_id=QTLWLCAG7MGVSYHG6BH65GPBIBEVEYRU7J56UAY4TC763PRSJ3MA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.761410337 (360h0m28.22350282s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGB2JAC3CMNKBE24VL6R2YS5ZQXJ6PREJ4UHMM3L4CUTQCJ4J4BA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.761723053 (360h0m28.223815526s) [noise=true , address=145.79.227.19:4079 , piece_id=HC3FC7XPGUTY7QANZD4FPRDXO7YLQIFKDSTVUHLOYLX5WG5EVVDQ, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.762034274 (360h0m28.224126747s) [noise=true , address=77.90.31.12:4589 , piece_id=4CASAPHOLRH7YGEMUJB33Z435N7SBBDHM7WOXEF7VXWEXCFNFAVA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.762300682 (360h0m28.224393155s) [noise=true , address=145.79.227.17:4971 , piece_id=G2JMMSEDTDJLWLKEMXODXY7DM3OIOTNDTRELBN3CZ4ICGGWJI73Q, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.762587067 (360h0m28.22467954s) [noise=true , address=145.79.227.15:4875 , piece_id=VX6S33AND7I7P6XONK575ISXAHKICATJ7NYSTPDAQ64CQJRDHFRA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.762857192 (360h0m28.224949675s) [noise=true , address=77.90.31.17:4771 , piece_id=WO645Q2EACGUPHLYDUK7245WORG6WMCOL36ZAZAULLZHRFM4SENA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:44.466626765 (360h0m29.928719248s) [upload=316734]: Next returned nil segment Oct 8 13:31:44.466648777 (360h0m29.92874125s) [upload=316734]: Waiting for error group managing segments... Oct 8 13:31:44.744695795 (360h0m30.206788268s) [noise=true , address=145.79.227.7:4295 , piece_id=SAF3CDB66F2REAWVSMRUNOK7B22CRHHANICMZT2RYTF6NVPOPOIA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.746695834 (360h0m30.208788317s) [noise=true , address=145.79.227.19:4079 , piece_id=HC3FC7XPGUTY7QANZD4FPRDXO7YLQIFKDSTVUHLOYLX5WG5EVVDQ, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.747509312 (360h0m30.209601795s) [noise=true , address=145.79.227.15:4875 , piece_id=VX6S33AND7I7P6XONK575ISXAHKICATJ7NYSTPDAQ64CQJRDHFRA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.754168252 (360h0m30.216260725s) [noise=true , address=145.79.227.4:4187 , piece_id=JHWEFNAVWBOSHXR6TEJJRQYWOCRPPMF4275PADBLE7DQ5GCEQ3RQ, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.760404758 (360h0m30.222497231s) [noise=true , address=145.79.227.11:4729 , piece_id=32VVCWGARNJZBETVKPQ4AYDPT7EZ4JAKNYSD7KBISQEJTKTBDRKA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.767948112 (360h0m30.230040595s) [noise=true , address=145.79.227.17:4971 , piece_id=G2JMMSEDTDJLWLKEMXODXY7DM3OIOTNDTRELBN3CZ4ICGGWJI73Q, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.797366731 (360h0m30.259459204s) [noise=true , address=145.79.227.5:4222 , piece_id=BNK2OZZ2I2BVGCW4Y2MUYXLEOCPTWJID37WR4ZIQ7RFXZGBPH6YQ, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.973460900 (360h0m30.435553373s) [noise=true , address=45.91.116.7:4009 , piece_id=O2TLURC6WLQQZIZBBNCEN2TRK42ZXQN2KGWGXC7QV34GFLCJMA5Q, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.985872144 (360h0m30.447964627s) [noise=true , address=45.91.116.5:4298 , piece_id=7E24KJ76NH6DE73HZRMIX2DENSTP2M6JFZEUSIHUWJNZN36TANTQ, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.059911309 (360h0m30.522003792s) [noise=true , address=45.91.116.8:4048 , piece_id=2BIYVGPYKNOCG4S3E4QTUWQUJVX64TEFXQV2NI2CTRMLLZX4KJNA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.060621962 (360h0m30.522714445s) [noise=true , address=77.90.31.17:4771 , piece_id=WO645Q2EACGUPHLYDUK7245WORG6WMCOL36ZAZAULLZHRFM4SENA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.075465818 (360h0m30.537558291s) [noise=true , address=77.90.31.16:4733 , piece_id=QEEUYHGBOH7SVXDSZL3GEN6HTZQDZJEDYVPJTXM3HR7SERMVRV5A, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.105250084 (360h0m30.567342557s) [noise=true , address=77.90.31.13:4626 , piece_id=6LKA3AE6QYGFCE6ZKBEXNEODMECL4LMP6ARE5OP7RCOZQF2KZGRA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.105305460 (360h0m30.567397933s) [seg_pos={0 0}, upload=316734]: Segment reached optimal threshold of 13 pieces. Oct 8 13:31:45.105442179 (360h0m30.567534652s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGB2JAC3CMNKBE24VL6R2YS5ZQXJ6PREJ4UHMM3L4CUTQCJ4J4BA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Oct 8 13:31:45.105449103 (360h0m30.567541576s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGB2JAC3CMNKBE24VL6R2YS5ZQXJ6PREJ4UHMM3L4CUTQCJ4J4BA, seg_pos={0 0}, upload=316734]: Long tail cancellation due to optimal threshold Oct 8 13:31:45.105522753 (360h0m30.567615236s) [noise=true , address=77.90.31.12:4589 , piece_id=4CASAPHOLRH7YGEMUJB33Z435N7SBBDHM7WOXEF7VXWEXCFNFAVA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Oct 8 13:31:45.105528474 (360h0m30.567620957s) [noise=true , address=77.90.31.12:4589 , piece_id=4CASAPHOLRH7YGEMUJB33Z435N7SBBDHM7WOXEF7VXWEXCFNFAVA, seg_pos={0 0}, upload=316734]: Long tail cancellation due to optimal threshold Oct 8 13:31:45.105568289 (360h0m30.567660763s) [noise=true , address=77.90.31.14:4662 , piece_id=QTLWLCAG7MGVSYHG6BH65GPBIBEVEYRU7J56UAY4TC763PRSJ3MA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 13:31:45.105588929 (360h0m30.567681402s) [noise=true , address=77.90.31.14:4662 , piece_id=QTLWLCAG7MGVSYHG6BH65GPBIBEVEYRU7J56UAY4TC763PRSJ3MA, seg_pos={0 0}, upload=316734]: Long tail cancellation due to optimal threshold Oct 8 13:31:45.105621772 (360h0m30.567714255s) [seg_pos={0 0}, upload=316734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:31:45.105706332 (360h0m30.567798815s) [upload=316734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:31:45.105713526 (360h0m30.567805999s) [upload=316734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:31:45.548510072 (360h0m31.010602555s) [upload=316734]: Done uploading segments... Oct 8 13:31:45.548533807 (360h0m31.01062629s) [upload=316734]: Upload finished. err: <nil> Duration: 27.063651556s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X........XX..XX...X.................X...................X.] duration: 28.398193552s FAIL (2025-10-08 13:32:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 2BIYVGPYKNOCG4S3E4QTUWQUJVX64TEFXQV2NI2CTRMLLZX4KJNA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: O2TLURC6WLQQZIZBBNCEN2TRK42ZXQN2KGWGXC7QV34GFLCJMA5Q) context deadline exceeded ======LOGS====== Oct 8 13:31:42.509127899 (360h0m27.971220372s) [upload=316734]: Starting upload Oct 8 13:31:42.509230875 (360h0m27.971323358s) [upload=316734]: Done starting upload Oct 8 13:31:42.509359320 (360h0m27.971451813s) [upload=316734]: Uploading segments... Oct 8 13:31:42.509483116 (360h0m27.971575599s) [upload=316734]: Got next segment. Inline: false Oct 8 13:31:42.509529254 (360h0m27.971621737s) [upload=316734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:31:42.509534975 (360h0m27.971627448s) [upload=316734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:31:42.757350406 (360h0m28.219442889s) [seg_pos={0 0}, upload=316734]: Begin upload segment... Oct 8 13:31:42.757395101 (360h0m28.219487584s) [seg_pos={0 0}, upload=316734]: Done begin upload segment. Oct 8 13:31:42.757436630 (360h0m28.219529113s) [noise=true , address=45.91.116.7:4009 , piece_id=O2TLURC6WLQQZIZBBNCEN2TRK42ZXQN2KGWGXC7QV34GFLCJMA5Q, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.758244649 (360h0m28.220337132s) [noise=true , address=145.79.227.4:4187 , piece_id=JHWEFNAVWBOSHXR6TEJJRQYWOCRPPMF4275PADBLE7DQ5GCEQ3RQ, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.758610716 (360h0m28.220703199s) [noise=true , address=45.91.116.5:4298 , piece_id=7E24KJ76NH6DE73HZRMIX2DENSTP2M6JFZEUSIHUWJNZN36TANTQ, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.758956725 (360h0m28.221049198s) [noise=true , address=145.79.227.7:4295 , piece_id=SAF3CDB66F2REAWVSMRUNOK7B22CRHHANICMZT2RYTF6NVPOPOIA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.759373288 (360h0m28.221465771s) [noise=true , address=145.79.227.11:4729 , piece_id=32VVCWGARNJZBETVKPQ4AYDPT7EZ4JAKNYSD7KBISQEJTKTBDRKA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.759845327 (360h0m28.22193781s) [noise=true , address=77.90.31.16:4733 , piece_id=QEEUYHGBOH7SVXDSZL3GEN6HTZQDZJEDYVPJTXM3HR7SERMVRV5A, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.760140038 (360h0m28.222232511s) [noise=true , address=145.79.227.5:4222 , piece_id=BNK2OZZ2I2BVGCW4Y2MUYXLEOCPTWJID37WR4ZIQ7RFXZGBPH6YQ, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.760506827 (360h0m28.22259931s) [noise=true , address=45.91.116.8:4048 , piece_id=2BIYVGPYKNOCG4S3E4QTUWQUJVX64TEFXQV2NI2CTRMLLZX4KJNA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.760848366 (360h0m28.222940849s) [noise=true , address=77.90.31.13:4626 , piece_id=6LKA3AE6QYGFCE6ZKBEXNEODMECL4LMP6ARE5OP7RCOZQF2KZGRA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.761135193 (360h0m28.223227666s) [noise=true , address=77.90.31.14:4662 , piece_id=QTLWLCAG7MGVSYHG6BH65GPBIBEVEYRU7J56UAY4TC763PRSJ3MA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.761410337 (360h0m28.22350282s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGB2JAC3CMNKBE24VL6R2YS5ZQXJ6PREJ4UHMM3L4CUTQCJ4J4BA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.761723053 (360h0m28.223815526s) [noise=true , address=145.79.227.19:4079 , piece_id=HC3FC7XPGUTY7QANZD4FPRDXO7YLQIFKDSTVUHLOYLX5WG5EVVDQ, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.762034274 (360h0m28.224126747s) [noise=true , address=77.90.31.12:4589 , piece_id=4CASAPHOLRH7YGEMUJB33Z435N7SBBDHM7WOXEF7VXWEXCFNFAVA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.762300682 (360h0m28.224393155s) [noise=true , address=145.79.227.17:4971 , piece_id=G2JMMSEDTDJLWLKEMXODXY7DM3OIOTNDTRELBN3CZ4ICGGWJI73Q, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.762587067 (360h0m28.22467954s) [noise=true , address=145.79.227.15:4875 , piece_id=VX6S33AND7I7P6XONK575ISXAHKICATJ7NYSTPDAQ64CQJRDHFRA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.762857192 (360h0m28.224949675s) [noise=true , address=77.90.31.17:4771 , piece_id=WO645Q2EACGUPHLYDUK7245WORG6WMCOL36ZAZAULLZHRFM4SENA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:44.466626765 (360h0m29.928719248s) [upload=316734]: Next returned nil segment Oct 8 13:31:44.466648777 (360h0m29.92874125s) [upload=316734]: Waiting for error group managing segments... Oct 8 13:31:44.744695795 (360h0m30.206788268s) [noise=true , address=145.79.227.7:4295 , piece_id=SAF3CDB66F2REAWVSMRUNOK7B22CRHHANICMZT2RYTF6NVPOPOIA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.746695834 (360h0m30.208788317s) [noise=true , address=145.79.227.19:4079 , piece_id=HC3FC7XPGUTY7QANZD4FPRDXO7YLQIFKDSTVUHLOYLX5WG5EVVDQ, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.747509312 (360h0m30.209601795s) [noise=true , address=145.79.227.15:4875 , piece_id=VX6S33AND7I7P6XONK575ISXAHKICATJ7NYSTPDAQ64CQJRDHFRA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.754168252 (360h0m30.216260725s) [noise=true , address=145.79.227.4:4187 , piece_id=JHWEFNAVWBOSHXR6TEJJRQYWOCRPPMF4275PADBLE7DQ5GCEQ3RQ, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.760404758 (360h0m30.222497231s) [noise=true , address=145.79.227.11:4729 , piece_id=32VVCWGARNJZBETVKPQ4AYDPT7EZ4JAKNYSD7KBISQEJTKTBDRKA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.767948112 (360h0m30.230040595s) [noise=true , address=145.79.227.17:4971 , piece_id=G2JMMSEDTDJLWLKEMXODXY7DM3OIOTNDTRELBN3CZ4ICGGWJI73Q, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.797366731 (360h0m30.259459204s) [noise=true , address=145.79.227.5:4222 , piece_id=BNK2OZZ2I2BVGCW4Y2MUYXLEOCPTWJID37WR4ZIQ7RFXZGBPH6YQ, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.973460900 (360h0m30.435553373s) [noise=true , address=45.91.116.7:4009 , piece_id=O2TLURC6WLQQZIZBBNCEN2TRK42ZXQN2KGWGXC7QV34GFLCJMA5Q, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.985872144 (360h0m30.447964627s) [noise=true , address=45.91.116.5:4298 , piece_id=7E24KJ76NH6DE73HZRMIX2DENSTP2M6JFZEUSIHUWJNZN36TANTQ, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.059911309 (360h0m30.522003792s) [noise=true , address=45.91.116.8:4048 , piece_id=2BIYVGPYKNOCG4S3E4QTUWQUJVX64TEFXQV2NI2CTRMLLZX4KJNA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.060621962 (360h0m30.522714445s) [noise=true , address=77.90.31.17:4771 , piece_id=WO645Q2EACGUPHLYDUK7245WORG6WMCOL36ZAZAULLZHRFM4SENA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.075465818 (360h0m30.537558291s) [noise=true , address=77.90.31.16:4733 , piece_id=QEEUYHGBOH7SVXDSZL3GEN6HTZQDZJEDYVPJTXM3HR7SERMVRV5A, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.105250084 (360h0m30.567342557s) [noise=true , address=77.90.31.13:4626 , piece_id=6LKA3AE6QYGFCE6ZKBEXNEODMECL4LMP6ARE5OP7RCOZQF2KZGRA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.105305460 (360h0m30.567397933s) [seg_pos={0 0}, upload=316734]: Segment reached optimal threshold of 13 pieces. Oct 8 13:31:45.105442179 (360h0m30.567534652s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGB2JAC3CMNKBE24VL6R2YS5ZQXJ6PREJ4UHMM3L4CUTQCJ4J4BA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Oct 8 13:31:45.105449103 (360h0m30.567541576s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGB2JAC3CMNKBE24VL6R2YS5ZQXJ6PREJ4UHMM3L4CUTQCJ4J4BA, seg_pos={0 0}, upload=316734]: Long tail cancellation due to optimal threshold Oct 8 13:31:45.105522753 (360h0m30.567615236s) [noise=true , address=77.90.31.12:4589 , piece_id=4CASAPHOLRH7YGEMUJB33Z435N7SBBDHM7WOXEF7VXWEXCFNFAVA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Oct 8 13:31:45.105528474 (360h0m30.567620957s) [noise=true , address=77.90.31.12:4589 , piece_id=4CASAPHOLRH7YGEMUJB33Z435N7SBBDHM7WOXEF7VXWEXCFNFAVA, seg_pos={0 0}, upload=316734]: Long tail cancellation due to optimal threshold Oct 8 13:31:45.105568289 (360h0m30.567660763s) [noise=true , address=77.90.31.14:4662 , piece_id=QTLWLCAG7MGVSYHG6BH65GPBIBEVEYRU7J56UAY4TC763PRSJ3MA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 13:31:45.105588929 (360h0m30.567681402s) [noise=true , address=77.90.31.14:4662 , piece_id=QTLWLCAG7MGVSYHG6BH65GPBIBEVEYRU7J56UAY4TC763PRSJ3MA, seg_pos={0 0}, upload=316734]: Long tail cancellation due to optimal threshold Oct 8 13:31:45.105621772 (360h0m30.567714255s) [seg_pos={0 0}, upload=316734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:31:45.105706332 (360h0m30.567798815s) [upload=316734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:31:45.105713526 (360h0m30.567805999s) [upload=316734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:31:45.548510072 (360h0m31.010602555s) [upload=316734]: Done uploading segments... Oct 8 13:31:45.548533807 (360h0m31.01062629s) [upload=316734]: Upload finished. err: <nil> Duration: 28.398225192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X..X........XX..XX...X.................X...................X] duration: 30.003139984s FAIL (2025-10-08 13:32:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 2BIYVGPYKNOCG4S3E4QTUWQUJVX64TEFXQV2NI2CTRMLLZX4KJNA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: O2TLURC6WLQQZIZBBNCEN2TRK42ZXQN2KGWGXC7QV34GFLCJMA5Q) context deadline exceeded ======LOGS====== Oct 8 13:31:42.509127899 (360h0m27.971220372s) [upload=316734]: Starting upload Oct 8 13:31:42.509230875 (360h0m27.971323358s) [upload=316734]: Done starting upload Oct 8 13:31:42.509359320 (360h0m27.971451813s) [upload=316734]: Uploading segments... Oct 8 13:31:42.509483116 (360h0m27.971575599s) [upload=316734]: Got next segment. Inline: false Oct 8 13:31:42.509529254 (360h0m27.971621737s) [upload=316734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:31:42.509534975 (360h0m27.971627448s) [upload=316734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:31:42.757350406 (360h0m28.219442889s) [seg_pos={0 0}, upload=316734]: Begin upload segment... Oct 8 13:31:42.757395101 (360h0m28.219487584s) [seg_pos={0 0}, upload=316734]: Done begin upload segment. Oct 8 13:31:42.757436630 (360h0m28.219529113s) [noise=true , address=45.91.116.7:4009 , piece_id=O2TLURC6WLQQZIZBBNCEN2TRK42ZXQN2KGWGXC7QV34GFLCJMA5Q, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.758244649 (360h0m28.220337132s) [noise=true , address=145.79.227.4:4187 , piece_id=JHWEFNAVWBOSHXR6TEJJRQYWOCRPPMF4275PADBLE7DQ5GCEQ3RQ, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.758610716 (360h0m28.220703199s) [noise=true , address=45.91.116.5:4298 , piece_id=7E24KJ76NH6DE73HZRMIX2DENSTP2M6JFZEUSIHUWJNZN36TANTQ, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.758956725 (360h0m28.221049198s) [noise=true , address=145.79.227.7:4295 , piece_id=SAF3CDB66F2REAWVSMRUNOK7B22CRHHANICMZT2RYTF6NVPOPOIA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.759373288 (360h0m28.221465771s) [noise=true , address=145.79.227.11:4729 , piece_id=32VVCWGARNJZBETVKPQ4AYDPT7EZ4JAKNYSD7KBISQEJTKTBDRKA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.759845327 (360h0m28.22193781s) [noise=true , address=77.90.31.16:4733 , piece_id=QEEUYHGBOH7SVXDSZL3GEN6HTZQDZJEDYVPJTXM3HR7SERMVRV5A, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.760140038 (360h0m28.222232511s) [noise=true , address=145.79.227.5:4222 , piece_id=BNK2OZZ2I2BVGCW4Y2MUYXLEOCPTWJID37WR4ZIQ7RFXZGBPH6YQ, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.760506827 (360h0m28.22259931s) [noise=true , address=45.91.116.8:4048 , piece_id=2BIYVGPYKNOCG4S3E4QTUWQUJVX64TEFXQV2NI2CTRMLLZX4KJNA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.760848366 (360h0m28.222940849s) [noise=true , address=77.90.31.13:4626 , piece_id=6LKA3AE6QYGFCE6ZKBEXNEODMECL4LMP6ARE5OP7RCOZQF2KZGRA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.761135193 (360h0m28.223227666s) [noise=true , address=77.90.31.14:4662 , piece_id=QTLWLCAG7MGVSYHG6BH65GPBIBEVEYRU7J56UAY4TC763PRSJ3MA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.761410337 (360h0m28.22350282s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGB2JAC3CMNKBE24VL6R2YS5ZQXJ6PREJ4UHMM3L4CUTQCJ4J4BA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.761723053 (360h0m28.223815526s) [noise=true , address=145.79.227.19:4079 , piece_id=HC3FC7XPGUTY7QANZD4FPRDXO7YLQIFKDSTVUHLOYLX5WG5EVVDQ, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.762034274 (360h0m28.224126747s) [noise=true , address=77.90.31.12:4589 , piece_id=4CASAPHOLRH7YGEMUJB33Z435N7SBBDHM7WOXEF7VXWEXCFNFAVA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.762300682 (360h0m28.224393155s) [noise=true , address=145.79.227.17:4971 , piece_id=G2JMMSEDTDJLWLKEMXODXY7DM3OIOTNDTRELBN3CZ4ICGGWJI73Q, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.762587067 (360h0m28.22467954s) [noise=true , address=145.79.227.15:4875 , piece_id=VX6S33AND7I7P6XONK575ISXAHKICATJ7NYSTPDAQ64CQJRDHFRA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:42.762857192 (360h0m28.224949675s) [noise=true , address=77.90.31.17:4771 , piece_id=WO645Q2EACGUPHLYDUK7245WORG6WMCOL36ZAZAULLZHRFM4SENA, seg_pos={0 0}, upload=316734]: Uploading piece... Oct 8 13:31:44.466626765 (360h0m29.928719248s) [upload=316734]: Next returned nil segment Oct 8 13:31:44.466648777 (360h0m29.92874125s) [upload=316734]: Waiting for error group managing segments... Oct 8 13:31:44.744695795 (360h0m30.206788268s) [noise=true , address=145.79.227.7:4295 , piece_id=SAF3CDB66F2REAWVSMRUNOK7B22CRHHANICMZT2RYTF6NVPOPOIA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.746695834 (360h0m30.208788317s) [noise=true , address=145.79.227.19:4079 , piece_id=HC3FC7XPGUTY7QANZD4FPRDXO7YLQIFKDSTVUHLOYLX5WG5EVVDQ, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.747509312 (360h0m30.209601795s) [noise=true , address=145.79.227.15:4875 , piece_id=VX6S33AND7I7P6XONK575ISXAHKICATJ7NYSTPDAQ64CQJRDHFRA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.754168252 (360h0m30.216260725s) [noise=true , address=145.79.227.4:4187 , piece_id=JHWEFNAVWBOSHXR6TEJJRQYWOCRPPMF4275PADBLE7DQ5GCEQ3RQ, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.760404758 (360h0m30.222497231s) [noise=true , address=145.79.227.11:4729 , piece_id=32VVCWGARNJZBETVKPQ4AYDPT7EZ4JAKNYSD7KBISQEJTKTBDRKA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.767948112 (360h0m30.230040595s) [noise=true , address=145.79.227.17:4971 , piece_id=G2JMMSEDTDJLWLKEMXODXY7DM3OIOTNDTRELBN3CZ4ICGGWJI73Q, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.797366731 (360h0m30.259459204s) [noise=true , address=145.79.227.5:4222 , piece_id=BNK2OZZ2I2BVGCW4Y2MUYXLEOCPTWJID37WR4ZIQ7RFXZGBPH6YQ, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.973460900 (360h0m30.435553373s) [noise=true , address=45.91.116.7:4009 , piece_id=O2TLURC6WLQQZIZBBNCEN2TRK42ZXQN2KGWGXC7QV34GFLCJMA5Q, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:44.985872144 (360h0m30.447964627s) [noise=true , address=45.91.116.5:4298 , piece_id=7E24KJ76NH6DE73HZRMIX2DENSTP2M6JFZEUSIHUWJNZN36TANTQ, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.059911309 (360h0m30.522003792s) [noise=true , address=45.91.116.8:4048 , piece_id=2BIYVGPYKNOCG4S3E4QTUWQUJVX64TEFXQV2NI2CTRMLLZX4KJNA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.060621962 (360h0m30.522714445s) [noise=true , address=77.90.31.17:4771 , piece_id=WO645Q2EACGUPHLYDUK7245WORG6WMCOL36ZAZAULLZHRFM4SENA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.075465818 (360h0m30.537558291s) [noise=true , address=77.90.31.16:4733 , piece_id=QEEUYHGBOH7SVXDSZL3GEN6HTZQDZJEDYVPJTXM3HR7SERMVRV5A, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.105250084 (360h0m30.567342557s) [noise=true , address=77.90.31.13:4626 , piece_id=6LKA3AE6QYGFCE6ZKBEXNEODMECL4LMP6ARE5OP7RCOZQF2KZGRA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: <nil> Oct 8 13:31:45.105305460 (360h0m30.567397933s) [seg_pos={0 0}, upload=316734]: Segment reached optimal threshold of 13 pieces. Oct 8 13:31:45.105442179 (360h0m30.567534652s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGB2JAC3CMNKBE24VL6R2YS5ZQXJ6PREJ4UHMM3L4CUTQCJ4J4BA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Oct 8 13:31:45.105449103 (360h0m30.567541576s) [noise=true , address=45.91.116.10:4083 , piece_id=ZGB2JAC3CMNKBE24VL6R2YS5ZQXJ6PREJ4UHMM3L4CUTQCJ4J4BA, seg_pos={0 0}, upload=316734]: Long tail cancellation due to optimal threshold Oct 8 13:31:45.105522753 (360h0m30.567615236s) [noise=true , address=77.90.31.12:4589 , piece_id=4CASAPHOLRH7YGEMUJB33Z435N7SBBDHM7WOXEF7VXWEXCFNFAVA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Oct 8 13:31:45.105528474 (360h0m30.567620957s) [noise=true , address=77.90.31.12:4589 , piece_id=4CASAPHOLRH7YGEMUJB33Z435N7SBBDHM7WOXEF7VXWEXCFNFAVA, seg_pos={0 0}, upload=316734]: Long tail cancellation due to optimal threshold Oct 8 13:31:45.105568289 (360h0m30.567660763s) [noise=true , address=77.90.31.14:4662 , piece_id=QTLWLCAG7MGVSYHG6BH65GPBIBEVEYRU7J56UAY4TC763PRSJ3MA, seg_pos={0 0}, upload=316734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 8 13:31:45.105588929 (360h0m30.567681402s) [noise=true , address=77.90.31.14:4662 , piece_id=QTLWLCAG7MGVSYHG6BH65GPBIBEVEYRU7J56UAY4TC763PRSJ3MA, seg_pos={0 0}, upload=316734]: Long tail cancellation due to optimal threshold Oct 8 13:31:45.105621772 (360h0m30.567714255s) [seg_pos={0 0}, upload=316734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:31:45.105706332 (360h0m30.567798815s) [upload=316734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:31:45.105713526 (360h0m30.567805999s) [upload=316734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:31:45.548510072 (360h0m31.010602555s) [upload=316734]: Done uploading segments... Oct 8 13:31:45.548533807 (360h0m31.01062629s) [upload=316734]: Upload finished. err: <nil> Duration: 30.003168287s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX..X........XX..XX...X.................X...................] duration: 7.848179709s FAIL (2025-10-08 13:30:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 7.848213585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XX..X........XX..XX...X.................X..................] duration: 6.487336238s FAIL (2025-10-08 13:30:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 6.487374039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.XX..X........XX..XX...X.................X.................] duration: 25.336489069s FAIL (2025-10-08 13:29:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 25.33651036s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.XX..X........XX..XX...X.................X................] duration: 29.173002111s FAIL (2025-10-08 13:29:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 29.173062328s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X.XX..X........XX..XX...X.................X...............] duration: 8.395096705s FAIL (2025-10-08 13:27:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 8.395144065s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X..X.XX..X........XX..XX...X.................X..............] duration: 20.520866065s FAIL (2025-10-08 13:27:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 20.520892957s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX..X.XX..X........XX..XX...X.................X.............] duration: 6.362129851s FAIL (2025-10-08 13:26:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 6.362154628s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XX..X.XX..X........XX..XX...X.................X............] duration: 23.875106814s FAIL (2025-10-08 13:25:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 23.87514086s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.XX..X.XX..X........XX..XX...X.................X...........] duration: 23.625686121s FAIL (2025-10-08 13:24:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 23.625720406s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XX.XX..X.XX..X........XX..XX...X.................X..........] duration: 6.507216023s FAIL (2025-10-08 13:24:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 6.50726203s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XXX.XX..X.XX..X........XX..XX...X.................X.........] duration: 19.733594193s FAIL (2025-10-08 13:24:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 19.733631394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XXX.XX..X.XX..X........XX..XX...X.................X........] duration: 6.04555164s FAIL (2025-10-08 13:22:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 6.045588501s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..XXX.XX..X.XX..X........XX..XX...X.................X.......] duration: 8.182142315s FAIL (2025-10-08 13:22:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 8.182172602s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..XXX.XX..X.XX..X........XX..XX...X.................X......] duration: 9.769118665s FAIL (2025-10-08 13:20:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 9.769146817s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX..XXX.XX..X.XX..X........XX..XX...X.................X.....] duration: 26.296576649s FAIL (2025-10-08 13:20:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 26.296612297s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX..XXX.XX..X.XX..X........XX..XX...X.................X....] duration: 9.433846011s FAIL (2025-10-08 13:18:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 9.433873143s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..XX..XXX.XX..X.XX..X........XX..XX...X.................X...] duration: 21.805434476s FAIL (2025-10-08 13:18:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 21.805490753s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...XX..XXX.XX..X.XX..X........XX..XX...X.................X..] duration: 26.778709479s FAIL (2025-10-08 13:17:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 26.778742371s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....XX..XXX.XX..X.XX..X........XX..XX...X.................X.] duration: 7.039481451s FAIL (2025-10-08 13:16:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 7.039503433s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X....XX..XXX.XX..X.XX..X........XX..XX...X.................X] duration: 30.003100733s FAIL (2025-10-08 13:16:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ) context deadline exceeded ======LOGS====== segments... Oct 8 13:15:43.737906161 (359h44m29.199998644s) [upload=316512]: Got next segment. Inline: false Oct 8 13:15:43.738003416 (359h44m29.200095899s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:15:43.738012323 (359h44m29.200104806s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:15:43.990452219 (359h44m29.452544692s) [seg_pos={0 0}, upload=316512]: Begin upload segment... Oct 8 13:15:43.990511813 (359h44m29.452604296s) [seg_pos={0 0}, upload=316512]: Done begin upload segment. Oct 8 13:15:43.990535377 (359h44m29.45262786s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.991699925 (359h44m29.453792408s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992434884 (359h44m29.454527367s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.992870835 (359h44m29.454963318s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993277068 (359h44m29.455369541s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.993686798 (359h44m29.455779281s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.995317834 (359h44m29.457410317s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.996986151 (359h44m29.459078624s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998596908 (359h44m29.460689391s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:43.998802530 (359h44m29.460895013s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.000336531 (359h44m29.462429014s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001560741 (359h44m29.463653215s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.001899787 (359h44m29.46399228s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.003593924 (359h44m29.465686417s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.005148885 (359h44m29.467241378s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:44.006064438 (359h44m29.468156911s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Uploading piece... Oct 8 13:15:45.698569147 (359h44m31.16066162s) [upload=316512]: Next returned nil segment Oct 8 13:15:45.698591450 (359h44m31.160683933s) [upload=316512]: Waiting for error group managing segments... Oct 8 13:15:45.977886073 (359h44m31.439978556s) [noise=true , address=145.79.227.11:4730 , piece_id=SNLWGHUAFR7KCG655ISDVCLWXAQBSHG5G7CDVSUOLRTXPJIE4GJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.981307979 (359h44m31.443400462s) [noise=true , address=145.79.227.13:4802 , piece_id=MQFUK44H6HDCEBB6GXR2ISBSP4SKG67TIYQTKZBB5ID2YQQWREVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.991981272 (359h44m31.454073755s) [noise=true , address=145.79.227.17:4972 , piece_id=VJSG2PZTXBWZOSMXD6U4LCIOXTNYH2VVGLWLCZOYCH4PUEOJ5JVQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992526140 (359h44m31.454618613s) [noise=true , address=145.79.227.5:4221 , piece_id=7WBMFGLCYPK6KIGUUO5HE5CV2HRP7NUVRRTYQEGQOCYLIQMWCUWQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:45.992849105 (359h44m31.454941618s) [noise=true , address=145.79.227.5:4222 , piece_id=HNXALNA6Y22MAI3MPF4RXIIKVS47EGDM76IXE7Z4IYR75UAL442Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.002446380 (359h44m31.464538863s) [noise=true , address=145.79.227.15:4876 , piece_id=SNGEEEVBAXJ356ZQCQONMBK6APWMDUQNJUJH6XMJ6XNPYWJPLC5A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013509196 (359h44m31.475601669s) [noise=true , address=145.79.227.3:4151 , piece_id=WQUVUMOECVMJJ6KRMWKOUSFQQWCWE6Q2OKH2KRE5LS4MYS5DR6ZQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.013771516 (359h44m31.475863999s) [noise=true , address=145.79.227.14:4838 , piece_id=2ACRGCRAFRSHZ6MQHODGB7J4WW35CGOWAQHOI7P6HQOFOU7CY6IQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.028841241 (359h44m31.490933724s) [noise=true , address=145.79.227.6:4258 , piece_id=T6G74QHZZRFYSCHDRVNUFVYQQFIMOSSYHRYC4XTPKNE633ZEV6WA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.218672048 (359h44m31.680764521s) [noise=true , address=45.91.116.13:4153 , piece_id=U4APLXUMVCQR5YS7TWWZMRGJOPLVUBZER3R5QHLNQVJPHVDVVIFA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.221575347 (359h44m31.68366782s) [noise=true , address=45.91.116.9:4262 , piece_id=CK6QIH36QPVBUQO3ZIILYVFBCAM6ZVJP3MWM42B4S6I5KFOS6YCQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.365061824 (359h44m31.827154307s) [noise=true , address=77.90.31.11:4555 , piece_id=L7D7ULZEYTPYRJILTYWMBRAU5SWLI7IN74EMRY7BJF4YDID5IJJA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584107391 (359h44m32.046199874s) [noise=true , address=45.91.116.12:4227 , piece_id=V2UBGVRGU36PUHYW5E65UAJVIQUQL2W7OG3ZBTCJKEFB4XOWBRFQ, seg_pos={0 0}, upload=316512]: Done uploading piece. err: <nil> Oct 8 13:15:46.584139963 (359h44m32.046232446s) [seg_pos={0 0}, upload=316512]: Segment reached optimal threshold of 13 pieces. Oct 8 13:15:46.584330124 (359h44m32.046422598s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Oct 8 13:15:46.584336597 (359h44m32.04642907s) [noise=true , address=77.90.31.16:4735 , piece_id=ASSSJDURF7GCPHDXHU3W5FIR5CZ57JEDPWDSXFY6Y47O2U2Z246A, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584346104 (359h44m32.046438588s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36982->77.90.31.19:4842: use of closed network connection Oct 8 13:15:46.584355463 (359h44m32.046447946s) [noise=true , address=77.90.31.19:4842 , piece_id=BPS5T3MNL4TM62B4XWHHZY7CMIQSWBOJZ72FOC5OQZ4Q4EQMYBEA, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584432690 (359h44m32.046525173s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53848->45.91.116.13:4154: use of closed network connection Oct 8 13:15:46.584440965 (359h44m32.046533438s) [noise=true , address=45.91.116.13:4154 , piece_id=GBB2EPSAHOXTD7R5UADBTY2HKPD5F4CIM4ZP6HEX6ZCEYHBJMC7Q, seg_pos={0 0}, upload=316512]: Long tail cancellation due to optimal threshold Oct 8 13:15:46.584460804 (359h44m32.046553287s) [seg_pos={0 0}, upload=316512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:15:46.584561515 (359h44m32.046653989s) [upload=316512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:15:46.584568579 (359h44m32.046661062s) [upload=316512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:15:46.908913205 (359h44m32.371005689s) [upload=316512]: Done uploading segments... Oct 8 13:15:46.908937631 (359h44m32.371030104s) [upload=316512]: Upload finished. err: <nil> Duration: 30.003125991s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X....XX..XXX.XX..X.XX..X........XX..XX...X.................] duration: 24.168500919s FAIL (2025-10-08 13:14:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ct 8 13:01:57.986076116 (359h30m43.448168599s) [upload=316311]: Starting upload Oct 8 13:01:57.986153663 (359h30m43.448246146s) [upload=316311]: Done starting upload Oct 8 13:01:57.986259396 (359h30m43.448351879s) [upload=316311]: Uploading segments... Oct 8 13:01:57.986408610 (359h30m43.448501083s) [upload=316311]: Got next segment. Inline: false Oct 8 13:01:57.986445650 (359h30m43.448538123s) [upload=316311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:01:57.986451090 (359h30m43.448543563s) [upload=316311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:01:58.252449976 (359h30m43.714542469s) [seg_pos={0 0}, upload=316311]: Begin upload segment... Oct 8 13:01:58.252567730 (359h30m43.714660213s) [seg_pos={0 0}, upload=316311]: Done begin upload segment. Oct 8 13:01:58.252603878 (359h30m43.714696361s) [noise=true , address=45.91.116.12:4225 , piece_id=6WTRWQ625W6WUV2DS27PNLPEWPPSR52H3YFOT6SCMG6TXP6W3J2A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.253640212 (359h30m43.715732705s) [noise=true , address=45.91.116.6:4975 , piece_id=P5JV6USSWDB335IRLXXCWMJEHYBUDN6KKCYSTJRWDLUEJEOY7RRA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.254216099 (359h30m43.716308582s) [noise=true , address=145.79.227.18:4005 , piece_id=6IB6TFKCAM7BEUQAMJESVPYV4HNGFZXV7R5YBT5NOM3IJKRBIRPQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.254793638 (359h30m43.716886111s) [noise=true , address=45.91.116.6:4973 , piece_id=GW7S6KTE5DLEUMUAEJE27VOFZY5YSTLOPJRCMKRZJCO3WCTQIJEQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.255413679 (359h30m43.717506152s) [noise=true , address=45.91.116.7:4009 , piece_id=HUHKWUSQB5LGFWPQZHHS6NTISCESL4WCZ5E5XJCDHK55FBWB6XXA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.255989366 (359h30m43.718081849s) [noise=true , address=77.90.31.10:4519 , piece_id=GVBAGJMLRINBGUGLR7K5U7U5UDAUO2TDHVQU3YOGH4LUMOZUJP4A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.256475141 (359h30m43.718567614s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.256916521 (359h30m43.719009005s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.257649487 (359h30m43.71974197s) [noise=true , address=45.91.116.14:4191 , piece_id=OB5O5DHWAVFLR64FIY2JMETOA6YBVHQZ7X623XG37FISJBLWQKKQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.258326596 (359h30m43.720419069s) [noise=true , address=145.79.227.13:4804 , piece_id=5CIOSYYYG2YEYYDM44JAGWEONLRJU2VJOCR6SA3TIIRB4DCBOSVQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.258976404 (359h30m43.721068887s) [noise=true , address=45.91.116.8:4048 , piece_id=F56BMKSENGC2HXMUOATBI3GSNPPCVVCCVEEQEZVEDI7QSTVE2YHQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.259581405 (359h30m43.721673898s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.260322587 (359h30m43.72241507s) [noise=true , address=77.90.31.19:4842 , piece_id=BTCUIS4IDUF5M4JA4HU6NIE3Q5LRX7ORPAM6XB534VPPARXLT7LQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.260930324 (359h30m43.723022807s) [noise=true , address=145.79.227.5:4223 , piece_id=O2VCLWGDNEN4C57VGXSL6QTC7Q676LALQI5X2CWPWFZBEN3XAHLA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.261489258 (359h30m43.723581741s) [noise=true , address=145.79.227.4:4187 , piece_id=BW3KBLEUESGC6EBPZPI3GJ3XZYAENVWANQT6JFHHPWCUKSKHTA5Q, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.262275075 (359h30m43.724367548s) [noise=true , address=145.79.227.3:4150 , piece_id=QG7IGRYT5SSGZI3A7FZGJ2DXAJU5RQVZSDBJR3BUQCV76NOPPQ6A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:59.950872797 (359h30m45.41296528s) [upload=316311]: Next returned nil segment Oct 8 13:01:59.950896232 (359h30m45.412988725s) [upload=316311]: Waiting for error group managing segments... Oct 8 13:02:00.230315202 (359h30m45.692407685s) [noise=true , address=145.79.227.4:4187 , piece_id=BW3KBLEUESGC6EBPZPI3GJ3XZYAENVWANQT6JFHHPWCUKSKHTA5Q, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.249571069 (359h30m45.711663542s) [noise=true , address=145.79.227.5:4223 , piece_id=O2VCLWGDNEN4C57VGXSL6QTC7Q676LALQI5X2CWPWFZBEN3XAHLA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.255522231 (359h30m45.717614724s) [noise=true , address=145.79.227.3:4150 , piece_id=QG7IGRYT5SSGZI3A7FZGJ2DXAJU5RQVZSDBJR3BUQCV76NOPPQ6A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.259550322 (359h30m45.721642805s) [noise=true , address=145.79.227.18:4005 , piece_id=6IB6TFKCAM7BEUQAMJESVPYV4HNGFZXV7R5YBT5NOM3IJKRBIRPQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.261539359 (359h30m45.723631832s) [noise=true , address=145.79.227.13:4804 , piece_id=5CIOSYYYG2YEYYDM44JAGWEONLRJU2VJOCR6SA3TIIRB4DCBOSVQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.455584772 (359h30m45.917677255s) [noise=true , address=45.91.116.6:4973 , piece_id=GW7S6KTE5DLEUMUAEJE27VOFZY5YSTLOPJRCMKRZJCO3WCTQIJEQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.526039842 (359h30m45.988132325s) [noise=true , address=77.90.31.10:4519 , piece_id=GVBAGJMLRINBGUGLR7K5U7U5UDAUO2TDHVQU3YOGH4LUMOZUJP4A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.533295910 (359h30m45.995388403s) [noise=true , address=77.90.31.19:4842 , piece_id=BTCUIS4IDUF5M4JA4HU6NIE3Q5LRX7ORPAM6XB534VPPARXLT7LQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.542682743 (359h30m46.004775216s) [noise=true , address=45.91.116.14:4191 , piece_id=OB5O5DHWAVFLR64FIY2JMETOA6YBVHQZ7X623XG37FISJBLWQKKQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.613214860 (359h30m46.075307343s) [noise=true , address=45.91.116.8:4048 , piece_id=F56BMKSENGC2HXMUOATBI3GSNPPCVVCCVEEQEZVEDI7QSTVE2YHQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.820474816 (359h30m46.282567299s) [noise=true , address=45.91.116.6:4975 , piece_id=P5JV6USSWDB335IRLXXCWMJEHYBUDN6KKCYSTJRWDLUEJEOY7RRA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.831238041 (359h30m46.293330524s) [noise=true , address=45.91.116.7:4009 , piece_id=HUHKWUSQB5LGFWPQZHHS6NTISCESL4WCZ5E5XJCDHK55FBWB6XXA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.849605917 (359h30m46.3116984s) [noise=true , address=45.91.116.12:4225 , piece_id=6WTRWQ625W6WUV2DS27PNLPEWPPSR52H3YFOT6SCMG6TXP6W3J2A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.849649890 (359h30m46.311742373s) [seg_pos={0 0}, upload=316311]: Segment reached optimal threshold of 13 pieces. Oct 8 13:02:00.849835444 (359h30m46.311927917s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection Oct 8 13:02:00.849844051 (359h30m46.311936524s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849879067 (359h30m46.31197154s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Oct 8 13:02:00.849888184 (359h30m46.311980657s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849927940 (359h30m46.312020413s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Oct 8 13:02:00.849934363 (359h30m46.312026836s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849955342 (359h30m46.312047825s) [seg_pos={0 0}, upload=316311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:02:00.850025265 (359h30m46.312117748s) [upload=316311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:02:00.850030355 (359h30m46.312122828s) [upload=316311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:02:01.159121227 (359h30m46.62121371s) [upload=316311]: Done uploading segments... Oct 8 13:02:01.159145763 (359h30m46.621238246s) [upload=316311]: Upload finished. err: <nil> Duration: 24.168530385s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X....XX..XXX.XX..X.XX..X........XX..XX...X................] duration: 6.107501514s FAIL (2025-10-08 13:14:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ct 8 13:01:57.986076116 (359h30m43.448168599s) [upload=316311]: Starting upload Oct 8 13:01:57.986153663 (359h30m43.448246146s) [upload=316311]: Done starting upload Oct 8 13:01:57.986259396 (359h30m43.448351879s) [upload=316311]: Uploading segments... Oct 8 13:01:57.986408610 (359h30m43.448501083s) [upload=316311]: Got next segment. Inline: false Oct 8 13:01:57.986445650 (359h30m43.448538123s) [upload=316311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:01:57.986451090 (359h30m43.448543563s) [upload=316311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:01:58.252449976 (359h30m43.714542469s) [seg_pos={0 0}, upload=316311]: Begin upload segment... Oct 8 13:01:58.252567730 (359h30m43.714660213s) [seg_pos={0 0}, upload=316311]: Done begin upload segment. Oct 8 13:01:58.252603878 (359h30m43.714696361s) [noise=true , address=45.91.116.12:4225 , piece_id=6WTRWQ625W6WUV2DS27PNLPEWPPSR52H3YFOT6SCMG6TXP6W3J2A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.253640212 (359h30m43.715732705s) [noise=true , address=45.91.116.6:4975 , piece_id=P5JV6USSWDB335IRLXXCWMJEHYBUDN6KKCYSTJRWDLUEJEOY7RRA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.254216099 (359h30m43.716308582s) [noise=true , address=145.79.227.18:4005 , piece_id=6IB6TFKCAM7BEUQAMJESVPYV4HNGFZXV7R5YBT5NOM3IJKRBIRPQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.254793638 (359h30m43.716886111s) [noise=true , address=45.91.116.6:4973 , piece_id=GW7S6KTE5DLEUMUAEJE27VOFZY5YSTLOPJRCMKRZJCO3WCTQIJEQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.255413679 (359h30m43.717506152s) [noise=true , address=45.91.116.7:4009 , piece_id=HUHKWUSQB5LGFWPQZHHS6NTISCESL4WCZ5E5XJCDHK55FBWB6XXA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.255989366 (359h30m43.718081849s) [noise=true , address=77.90.31.10:4519 , piece_id=GVBAGJMLRINBGUGLR7K5U7U5UDAUO2TDHVQU3YOGH4LUMOZUJP4A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.256475141 (359h30m43.718567614s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.256916521 (359h30m43.719009005s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.257649487 (359h30m43.71974197s) [noise=true , address=45.91.116.14:4191 , piece_id=OB5O5DHWAVFLR64FIY2JMETOA6YBVHQZ7X623XG37FISJBLWQKKQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.258326596 (359h30m43.720419069s) [noise=true , address=145.79.227.13:4804 , piece_id=5CIOSYYYG2YEYYDM44JAGWEONLRJU2VJOCR6SA3TIIRB4DCBOSVQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.258976404 (359h30m43.721068887s) [noise=true , address=45.91.116.8:4048 , piece_id=F56BMKSENGC2HXMUOATBI3GSNPPCVVCCVEEQEZVEDI7QSTVE2YHQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.259581405 (359h30m43.721673898s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.260322587 (359h30m43.72241507s) [noise=true , address=77.90.31.19:4842 , piece_id=BTCUIS4IDUF5M4JA4HU6NIE3Q5LRX7ORPAM6XB534VPPARXLT7LQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.260930324 (359h30m43.723022807s) [noise=true , address=145.79.227.5:4223 , piece_id=O2VCLWGDNEN4C57VGXSL6QTC7Q676LALQI5X2CWPWFZBEN3XAHLA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.261489258 (359h30m43.723581741s) [noise=true , address=145.79.227.4:4187 , piece_id=BW3KBLEUESGC6EBPZPI3GJ3XZYAENVWANQT6JFHHPWCUKSKHTA5Q, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.262275075 (359h30m43.724367548s) [noise=true , address=145.79.227.3:4150 , piece_id=QG7IGRYT5SSGZI3A7FZGJ2DXAJU5RQVZSDBJR3BUQCV76NOPPQ6A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:59.950872797 (359h30m45.41296528s) [upload=316311]: Next returned nil segment Oct 8 13:01:59.950896232 (359h30m45.412988725s) [upload=316311]: Waiting for error group managing segments... Oct 8 13:02:00.230315202 (359h30m45.692407685s) [noise=true , address=145.79.227.4:4187 , piece_id=BW3KBLEUESGC6EBPZPI3GJ3XZYAENVWANQT6JFHHPWCUKSKHTA5Q, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.249571069 (359h30m45.711663542s) [noise=true , address=145.79.227.5:4223 , piece_id=O2VCLWGDNEN4C57VGXSL6QTC7Q676LALQI5X2CWPWFZBEN3XAHLA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.255522231 (359h30m45.717614724s) [noise=true , address=145.79.227.3:4150 , piece_id=QG7IGRYT5SSGZI3A7FZGJ2DXAJU5RQVZSDBJR3BUQCV76NOPPQ6A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.259550322 (359h30m45.721642805s) [noise=true , address=145.79.227.18:4005 , piece_id=6IB6TFKCAM7BEUQAMJESVPYV4HNGFZXV7R5YBT5NOM3IJKRBIRPQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.261539359 (359h30m45.723631832s) [noise=true , address=145.79.227.13:4804 , piece_id=5CIOSYYYG2YEYYDM44JAGWEONLRJU2VJOCR6SA3TIIRB4DCBOSVQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.455584772 (359h30m45.917677255s) [noise=true , address=45.91.116.6:4973 , piece_id=GW7S6KTE5DLEUMUAEJE27VOFZY5YSTLOPJRCMKRZJCO3WCTQIJEQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.526039842 (359h30m45.988132325s) [noise=true , address=77.90.31.10:4519 , piece_id=GVBAGJMLRINBGUGLR7K5U7U5UDAUO2TDHVQU3YOGH4LUMOZUJP4A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.533295910 (359h30m45.995388403s) [noise=true , address=77.90.31.19:4842 , piece_id=BTCUIS4IDUF5M4JA4HU6NIE3Q5LRX7ORPAM6XB534VPPARXLT7LQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.542682743 (359h30m46.004775216s) [noise=true , address=45.91.116.14:4191 , piece_id=OB5O5DHWAVFLR64FIY2JMETOA6YBVHQZ7X623XG37FISJBLWQKKQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.613214860 (359h30m46.075307343s) [noise=true , address=45.91.116.8:4048 , piece_id=F56BMKSENGC2HXMUOATBI3GSNPPCVVCCVEEQEZVEDI7QSTVE2YHQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.820474816 (359h30m46.282567299s) [noise=true , address=45.91.116.6:4975 , piece_id=P5JV6USSWDB335IRLXXCWMJEHYBUDN6KKCYSTJRWDLUEJEOY7RRA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.831238041 (359h30m46.293330524s) [noise=true , address=45.91.116.7:4009 , piece_id=HUHKWUSQB5LGFWPQZHHS6NTISCESL4WCZ5E5XJCDHK55FBWB6XXA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.849605917 (359h30m46.3116984s) [noise=true , address=45.91.116.12:4225 , piece_id=6WTRWQ625W6WUV2DS27PNLPEWPPSR52H3YFOT6SCMG6TXP6W3J2A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.849649890 (359h30m46.311742373s) [seg_pos={0 0}, upload=316311]: Segment reached optimal threshold of 13 pieces. Oct 8 13:02:00.849835444 (359h30m46.311927917s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection Oct 8 13:02:00.849844051 (359h30m46.311936524s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849879067 (359h30m46.31197154s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Oct 8 13:02:00.849888184 (359h30m46.311980657s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849927940 (359h30m46.312020413s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Oct 8 13:02:00.849934363 (359h30m46.312026836s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849955342 (359h30m46.312047825s) [seg_pos={0 0}, upload=316311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:02:00.850025265 (359h30m46.312117748s) [upload=316311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:02:00.850030355 (359h30m46.312122828s) [upload=316311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:02:01.159121227 (359h30m46.62121371s) [upload=316311]: Done uploading segments... Oct 8 13:02:01.159145763 (359h30m46.621238246s) [upload=316311]: Upload finished. err: <nil> Duration: 6.107533003s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.137541895s FAIL (2025-10-08 13:14:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.137562765s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..X....XX..XXX.XX..X.XX..X........XX..XX...X...............] duration: 7.109843399s FAIL (2025-10-08 13:13:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ct 8 13:01:57.986076116 (359h30m43.448168599s) [upload=316311]: Starting upload Oct 8 13:01:57.986153663 (359h30m43.448246146s) [upload=316311]: Done starting upload Oct 8 13:01:57.986259396 (359h30m43.448351879s) [upload=316311]: Uploading segments... Oct 8 13:01:57.986408610 (359h30m43.448501083s) [upload=316311]: Got next segment. Inline: false Oct 8 13:01:57.986445650 (359h30m43.448538123s) [upload=316311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:01:57.986451090 (359h30m43.448543563s) [upload=316311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:01:58.252449976 (359h30m43.714542469s) [seg_pos={0 0}, upload=316311]: Begin upload segment... Oct 8 13:01:58.252567730 (359h30m43.714660213s) [seg_pos={0 0}, upload=316311]: Done begin upload segment. Oct 8 13:01:58.252603878 (359h30m43.714696361s) [noise=true , address=45.91.116.12:4225 , piece_id=6WTRWQ625W6WUV2DS27PNLPEWPPSR52H3YFOT6SCMG6TXP6W3J2A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.253640212 (359h30m43.715732705s) [noise=true , address=45.91.116.6:4975 , piece_id=P5JV6USSWDB335IRLXXCWMJEHYBUDN6KKCYSTJRWDLUEJEOY7RRA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.254216099 (359h30m43.716308582s) [noise=true , address=145.79.227.18:4005 , piece_id=6IB6TFKCAM7BEUQAMJESVPYV4HNGFZXV7R5YBT5NOM3IJKRBIRPQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.254793638 (359h30m43.716886111s) [noise=true , address=45.91.116.6:4973 , piece_id=GW7S6KTE5DLEUMUAEJE27VOFZY5YSTLOPJRCMKRZJCO3WCTQIJEQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.255413679 (359h30m43.717506152s) [noise=true , address=45.91.116.7:4009 , piece_id=HUHKWUSQB5LGFWPQZHHS6NTISCESL4WCZ5E5XJCDHK55FBWB6XXA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.255989366 (359h30m43.718081849s) [noise=true , address=77.90.31.10:4519 , piece_id=GVBAGJMLRINBGUGLR7K5U7U5UDAUO2TDHVQU3YOGH4LUMOZUJP4A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.256475141 (359h30m43.718567614s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.256916521 (359h30m43.719009005s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.257649487 (359h30m43.71974197s) [noise=true , address=45.91.116.14:4191 , piece_id=OB5O5DHWAVFLR64FIY2JMETOA6YBVHQZ7X623XG37FISJBLWQKKQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.258326596 (359h30m43.720419069s) [noise=true , address=145.79.227.13:4804 , piece_id=5CIOSYYYG2YEYYDM44JAGWEONLRJU2VJOCR6SA3TIIRB4DCBOSVQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.258976404 (359h30m43.721068887s) [noise=true , address=45.91.116.8:4048 , piece_id=F56BMKSENGC2HXMUOATBI3GSNPPCVVCCVEEQEZVEDI7QSTVE2YHQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.259581405 (359h30m43.721673898s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.260322587 (359h30m43.72241507s) [noise=true , address=77.90.31.19:4842 , piece_id=BTCUIS4IDUF5M4JA4HU6NIE3Q5LRX7ORPAM6XB534VPPARXLT7LQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.260930324 (359h30m43.723022807s) [noise=true , address=145.79.227.5:4223 , piece_id=O2VCLWGDNEN4C57VGXSL6QTC7Q676LALQI5X2CWPWFZBEN3XAHLA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.261489258 (359h30m43.723581741s) [noise=true , address=145.79.227.4:4187 , piece_id=BW3KBLEUESGC6EBPZPI3GJ3XZYAENVWANQT6JFHHPWCUKSKHTA5Q, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.262275075 (359h30m43.724367548s) [noise=true , address=145.79.227.3:4150 , piece_id=QG7IGRYT5SSGZI3A7FZGJ2DXAJU5RQVZSDBJR3BUQCV76NOPPQ6A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:59.950872797 (359h30m45.41296528s) [upload=316311]: Next returned nil segment Oct 8 13:01:59.950896232 (359h30m45.412988725s) [upload=316311]: Waiting for error group managing segments... Oct 8 13:02:00.230315202 (359h30m45.692407685s) [noise=true , address=145.79.227.4:4187 , piece_id=BW3KBLEUESGC6EBPZPI3GJ3XZYAENVWANQT6JFHHPWCUKSKHTA5Q, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.249571069 (359h30m45.711663542s) [noise=true , address=145.79.227.5:4223 , piece_id=O2VCLWGDNEN4C57VGXSL6QTC7Q676LALQI5X2CWPWFZBEN3XAHLA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.255522231 (359h30m45.717614724s) [noise=true , address=145.79.227.3:4150 , piece_id=QG7IGRYT5SSGZI3A7FZGJ2DXAJU5RQVZSDBJR3BUQCV76NOPPQ6A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.259550322 (359h30m45.721642805s) [noise=true , address=145.79.227.18:4005 , piece_id=6IB6TFKCAM7BEUQAMJESVPYV4HNGFZXV7R5YBT5NOM3IJKRBIRPQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.261539359 (359h30m45.723631832s) [noise=true , address=145.79.227.13:4804 , piece_id=5CIOSYYYG2YEYYDM44JAGWEONLRJU2VJOCR6SA3TIIRB4DCBOSVQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.455584772 (359h30m45.917677255s) [noise=true , address=45.91.116.6:4973 , piece_id=GW7S6KTE5DLEUMUAEJE27VOFZY5YSTLOPJRCMKRZJCO3WCTQIJEQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.526039842 (359h30m45.988132325s) [noise=true , address=77.90.31.10:4519 , piece_id=GVBAGJMLRINBGUGLR7K5U7U5UDAUO2TDHVQU3YOGH4LUMOZUJP4A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.533295910 (359h30m45.995388403s) [noise=true , address=77.90.31.19:4842 , piece_id=BTCUIS4IDUF5M4JA4HU6NIE3Q5LRX7ORPAM6XB534VPPARXLT7LQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.542682743 (359h30m46.004775216s) [noise=true , address=45.91.116.14:4191 , piece_id=OB5O5DHWAVFLR64FIY2JMETOA6YBVHQZ7X623XG37FISJBLWQKKQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.613214860 (359h30m46.075307343s) [noise=true , address=45.91.116.8:4048 , piece_id=F56BMKSENGC2HXMUOATBI3GSNPPCVVCCVEEQEZVEDI7QSTVE2YHQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.820474816 (359h30m46.282567299s) [noise=true , address=45.91.116.6:4975 , piece_id=P5JV6USSWDB335IRLXXCWMJEHYBUDN6KKCYSTJRWDLUEJEOY7RRA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.831238041 (359h30m46.293330524s) [noise=true , address=45.91.116.7:4009 , piece_id=HUHKWUSQB5LGFWPQZHHS6NTISCESL4WCZ5E5XJCDHK55FBWB6XXA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.849605917 (359h30m46.3116984s) [noise=true , address=45.91.116.12:4225 , piece_id=6WTRWQ625W6WUV2DS27PNLPEWPPSR52H3YFOT6SCMG6TXP6W3J2A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.849649890 (359h30m46.311742373s) [seg_pos={0 0}, upload=316311]: Segment reached optimal threshold of 13 pieces. Oct 8 13:02:00.849835444 (359h30m46.311927917s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection Oct 8 13:02:00.849844051 (359h30m46.311936524s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849879067 (359h30m46.31197154s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Oct 8 13:02:00.849888184 (359h30m46.311980657s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849927940 (359h30m46.312020413s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Oct 8 13:02:00.849934363 (359h30m46.312026836s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849955342 (359h30m46.312047825s) [seg_pos={0 0}, upload=316311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:02:00.850025265 (359h30m46.312117748s) [upload=316311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:02:00.850030355 (359h30m46.312122828s) [upload=316311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:02:01.159121227 (359h30m46.62121371s) [upload=316311]: Done uploading segments... Oct 8 13:02:01.159145763 (359h30m46.621238246s) [upload=316311]: Upload finished. err: <nil> Duration: 7.109877713s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.212309763s FAIL (2025-10-08 13:12:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.212338308s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..X....XX..XXX.XX..X.XX..X........XX..XX...X..............] duration: 9.498170921s FAIL (2025-10-08 13:12:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ct 8 13:01:57.986076116 (359h30m43.448168599s) [upload=316311]: Starting upload Oct 8 13:01:57.986153663 (359h30m43.448246146s) [upload=316311]: Done starting upload Oct 8 13:01:57.986259396 (359h30m43.448351879s) [upload=316311]: Uploading segments... Oct 8 13:01:57.986408610 (359h30m43.448501083s) [upload=316311]: Got next segment. Inline: false Oct 8 13:01:57.986445650 (359h30m43.448538123s) [upload=316311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:01:57.986451090 (359h30m43.448543563s) [upload=316311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:01:58.252449976 (359h30m43.714542469s) [seg_pos={0 0}, upload=316311]: Begin upload segment... Oct 8 13:01:58.252567730 (359h30m43.714660213s) [seg_pos={0 0}, upload=316311]: Done begin upload segment. Oct 8 13:01:58.252603878 (359h30m43.714696361s) [noise=true , address=45.91.116.12:4225 , piece_id=6WTRWQ625W6WUV2DS27PNLPEWPPSR52H3YFOT6SCMG6TXP6W3J2A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.253640212 (359h30m43.715732705s) [noise=true , address=45.91.116.6:4975 , piece_id=P5JV6USSWDB335IRLXXCWMJEHYBUDN6KKCYSTJRWDLUEJEOY7RRA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.254216099 (359h30m43.716308582s) [noise=true , address=145.79.227.18:4005 , piece_id=6IB6TFKCAM7BEUQAMJESVPYV4HNGFZXV7R5YBT5NOM3IJKRBIRPQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.254793638 (359h30m43.716886111s) [noise=true , address=45.91.116.6:4973 , piece_id=GW7S6KTE5DLEUMUAEJE27VOFZY5YSTLOPJRCMKRZJCO3WCTQIJEQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.255413679 (359h30m43.717506152s) [noise=true , address=45.91.116.7:4009 , piece_id=HUHKWUSQB5LGFWPQZHHS6NTISCESL4WCZ5E5XJCDHK55FBWB6XXA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.255989366 (359h30m43.718081849s) [noise=true , address=77.90.31.10:4519 , piece_id=GVBAGJMLRINBGUGLR7K5U7U5UDAUO2TDHVQU3YOGH4LUMOZUJP4A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.256475141 (359h30m43.718567614s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.256916521 (359h30m43.719009005s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.257649487 (359h30m43.71974197s) [noise=true , address=45.91.116.14:4191 , piece_id=OB5O5DHWAVFLR64FIY2JMETOA6YBVHQZ7X623XG37FISJBLWQKKQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.258326596 (359h30m43.720419069s) [noise=true , address=145.79.227.13:4804 , piece_id=5CIOSYYYG2YEYYDM44JAGWEONLRJU2VJOCR6SA3TIIRB4DCBOSVQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.258976404 (359h30m43.721068887s) [noise=true , address=45.91.116.8:4048 , piece_id=F56BMKSENGC2HXMUOATBI3GSNPPCVVCCVEEQEZVEDI7QSTVE2YHQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.259581405 (359h30m43.721673898s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.260322587 (359h30m43.72241507s) [noise=true , address=77.90.31.19:4842 , piece_id=BTCUIS4IDUF5M4JA4HU6NIE3Q5LRX7ORPAM6XB534VPPARXLT7LQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.260930324 (359h30m43.723022807s) [noise=true , address=145.79.227.5:4223 , piece_id=O2VCLWGDNEN4C57VGXSL6QTC7Q676LALQI5X2CWPWFZBEN3XAHLA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.261489258 (359h30m43.723581741s) [noise=true , address=145.79.227.4:4187 , piece_id=BW3KBLEUESGC6EBPZPI3GJ3XZYAENVWANQT6JFHHPWCUKSKHTA5Q, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.262275075 (359h30m43.724367548s) [noise=true , address=145.79.227.3:4150 , piece_id=QG7IGRYT5SSGZI3A7FZGJ2DXAJU5RQVZSDBJR3BUQCV76NOPPQ6A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:59.950872797 (359h30m45.41296528s) [upload=316311]: Next returned nil segment Oct 8 13:01:59.950896232 (359h30m45.412988725s) [upload=316311]: Waiting for error group managing segments... Oct 8 13:02:00.230315202 (359h30m45.692407685s) [noise=true , address=145.79.227.4:4187 , piece_id=BW3KBLEUESGC6EBPZPI3GJ3XZYAENVWANQT6JFHHPWCUKSKHTA5Q, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.249571069 (359h30m45.711663542s) [noise=true , address=145.79.227.5:4223 , piece_id=O2VCLWGDNEN4C57VGXSL6QTC7Q676LALQI5X2CWPWFZBEN3XAHLA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.255522231 (359h30m45.717614724s) [noise=true , address=145.79.227.3:4150 , piece_id=QG7IGRYT5SSGZI3A7FZGJ2DXAJU5RQVZSDBJR3BUQCV76NOPPQ6A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.259550322 (359h30m45.721642805s) [noise=true , address=145.79.227.18:4005 , piece_id=6IB6TFKCAM7BEUQAMJESVPYV4HNGFZXV7R5YBT5NOM3IJKRBIRPQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.261539359 (359h30m45.723631832s) [noise=true , address=145.79.227.13:4804 , piece_id=5CIOSYYYG2YEYYDM44JAGWEONLRJU2VJOCR6SA3TIIRB4DCBOSVQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.455584772 (359h30m45.917677255s) [noise=true , address=45.91.116.6:4973 , piece_id=GW7S6KTE5DLEUMUAEJE27VOFZY5YSTLOPJRCMKRZJCO3WCTQIJEQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.526039842 (359h30m45.988132325s) [noise=true , address=77.90.31.10:4519 , piece_id=GVBAGJMLRINBGUGLR7K5U7U5UDAUO2TDHVQU3YOGH4LUMOZUJP4A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.533295910 (359h30m45.995388403s) [noise=true , address=77.90.31.19:4842 , piece_id=BTCUIS4IDUF5M4JA4HU6NIE3Q5LRX7ORPAM6XB534VPPARXLT7LQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.542682743 (359h30m46.004775216s) [noise=true , address=45.91.116.14:4191 , piece_id=OB5O5DHWAVFLR64FIY2JMETOA6YBVHQZ7X623XG37FISJBLWQKKQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.613214860 (359h30m46.075307343s) [noise=true , address=45.91.116.8:4048 , piece_id=F56BMKSENGC2HXMUOATBI3GSNPPCVVCCVEEQEZVEDI7QSTVE2YHQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.820474816 (359h30m46.282567299s) [noise=true , address=45.91.116.6:4975 , piece_id=P5JV6USSWDB335IRLXXCWMJEHYBUDN6KKCYSTJRWDLUEJEOY7RRA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.831238041 (359h30m46.293330524s) [noise=true , address=45.91.116.7:4009 , piece_id=HUHKWUSQB5LGFWPQZHHS6NTISCESL4WCZ5E5XJCDHK55FBWB6XXA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.849605917 (359h30m46.3116984s) [noise=true , address=45.91.116.12:4225 , piece_id=6WTRWQ625W6WUV2DS27PNLPEWPPSR52H3YFOT6SCMG6TXP6W3J2A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.849649890 (359h30m46.311742373s) [seg_pos={0 0}, upload=316311]: Segment reached optimal threshold of 13 pieces. Oct 8 13:02:00.849835444 (359h30m46.311927917s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection Oct 8 13:02:00.849844051 (359h30m46.311936524s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849879067 (359h30m46.31197154s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Oct 8 13:02:00.849888184 (359h30m46.311980657s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849927940 (359h30m46.312020413s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Oct 8 13:02:00.849934363 (359h30m46.312026836s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849955342 (359h30m46.312047825s) [seg_pos={0 0}, upload=316311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:02:00.850025265 (359h30m46.312117748s) [upload=316311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:02:00.850030355 (359h30m46.312122828s) [upload=316311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:02:01.159121227 (359h30m46.62121371s) [upload=316311]: Done uploading segments... Oct 8 13:02:01.159145763 (359h30m46.621238246s) [upload=316311]: Upload finished. err: <nil> Duration: 9.498201388s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.310084894s FAIL (2025-10-08 13:12:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.310231865s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X..X....XX..XXX.XX..X.XX..X........XX..XX...X.............] duration: 9.222414637s FAIL (2025-10-08 13:11:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ct 8 13:01:57.986076116 (359h30m43.448168599s) [upload=316311]: Starting upload Oct 8 13:01:57.986153663 (359h30m43.448246146s) [upload=316311]: Done starting upload Oct 8 13:01:57.986259396 (359h30m43.448351879s) [upload=316311]: Uploading segments... Oct 8 13:01:57.986408610 (359h30m43.448501083s) [upload=316311]: Got next segment. Inline: false Oct 8 13:01:57.986445650 (359h30m43.448538123s) [upload=316311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 8 13:01:57.986451090 (359h30m43.448543563s) [upload=316311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 8 13:01:58.252449976 (359h30m43.714542469s) [seg_pos={0 0}, upload=316311]: Begin upload segment... Oct 8 13:01:58.252567730 (359h30m43.714660213s) [seg_pos={0 0}, upload=316311]: Done begin upload segment. Oct 8 13:01:58.252603878 (359h30m43.714696361s) [noise=true , address=45.91.116.12:4225 , piece_id=6WTRWQ625W6WUV2DS27PNLPEWPPSR52H3YFOT6SCMG6TXP6W3J2A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.253640212 (359h30m43.715732705s) [noise=true , address=45.91.116.6:4975 , piece_id=P5JV6USSWDB335IRLXXCWMJEHYBUDN6KKCYSTJRWDLUEJEOY7RRA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.254216099 (359h30m43.716308582s) [noise=true , address=145.79.227.18:4005 , piece_id=6IB6TFKCAM7BEUQAMJESVPYV4HNGFZXV7R5YBT5NOM3IJKRBIRPQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.254793638 (359h30m43.716886111s) [noise=true , address=45.91.116.6:4973 , piece_id=GW7S6KTE5DLEUMUAEJE27VOFZY5YSTLOPJRCMKRZJCO3WCTQIJEQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.255413679 (359h30m43.717506152s) [noise=true , address=45.91.116.7:4009 , piece_id=HUHKWUSQB5LGFWPQZHHS6NTISCESL4WCZ5E5XJCDHK55FBWB6XXA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.255989366 (359h30m43.718081849s) [noise=true , address=77.90.31.10:4519 , piece_id=GVBAGJMLRINBGUGLR7K5U7U5UDAUO2TDHVQU3YOGH4LUMOZUJP4A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.256475141 (359h30m43.718567614s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.256916521 (359h30m43.719009005s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.257649487 (359h30m43.71974197s) [noise=true , address=45.91.116.14:4191 , piece_id=OB5O5DHWAVFLR64FIY2JMETOA6YBVHQZ7X623XG37FISJBLWQKKQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.258326596 (359h30m43.720419069s) [noise=true , address=145.79.227.13:4804 , piece_id=5CIOSYYYG2YEYYDM44JAGWEONLRJU2VJOCR6SA3TIIRB4DCBOSVQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.258976404 (359h30m43.721068887s) [noise=true , address=45.91.116.8:4048 , piece_id=F56BMKSENGC2HXMUOATBI3GSNPPCVVCCVEEQEZVEDI7QSTVE2YHQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.259581405 (359h30m43.721673898s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.260322587 (359h30m43.72241507s) [noise=true , address=77.90.31.19:4842 , piece_id=BTCUIS4IDUF5M4JA4HU6NIE3Q5LRX7ORPAM6XB534VPPARXLT7LQ, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.260930324 (359h30m43.723022807s) [noise=true , address=145.79.227.5:4223 , piece_id=O2VCLWGDNEN4C57VGXSL6QTC7Q676LALQI5X2CWPWFZBEN3XAHLA, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.261489258 (359h30m43.723581741s) [noise=true , address=145.79.227.4:4187 , piece_id=BW3KBLEUESGC6EBPZPI3GJ3XZYAENVWANQT6JFHHPWCUKSKHTA5Q, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:58.262275075 (359h30m43.724367548s) [noise=true , address=145.79.227.3:4150 , piece_id=QG7IGRYT5SSGZI3A7FZGJ2DXAJU5RQVZSDBJR3BUQCV76NOPPQ6A, seg_pos={0 0}, upload=316311]: Uploading piece... Oct 8 13:01:59.950872797 (359h30m45.41296528s) [upload=316311]: Next returned nil segment Oct 8 13:01:59.950896232 (359h30m45.412988725s) [upload=316311]: Waiting for error group managing segments... Oct 8 13:02:00.230315202 (359h30m45.692407685s) [noise=true , address=145.79.227.4:4187 , piece_id=BW3KBLEUESGC6EBPZPI3GJ3XZYAENVWANQT6JFHHPWCUKSKHTA5Q, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.249571069 (359h30m45.711663542s) [noise=true , address=145.79.227.5:4223 , piece_id=O2VCLWGDNEN4C57VGXSL6QTC7Q676LALQI5X2CWPWFZBEN3XAHLA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.255522231 (359h30m45.717614724s) [noise=true , address=145.79.227.3:4150 , piece_id=QG7IGRYT5SSGZI3A7FZGJ2DXAJU5RQVZSDBJR3BUQCV76NOPPQ6A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.259550322 (359h30m45.721642805s) [noise=true , address=145.79.227.18:4005 , piece_id=6IB6TFKCAM7BEUQAMJESVPYV4HNGFZXV7R5YBT5NOM3IJKRBIRPQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.261539359 (359h30m45.723631832s) [noise=true , address=145.79.227.13:4804 , piece_id=5CIOSYYYG2YEYYDM44JAGWEONLRJU2VJOCR6SA3TIIRB4DCBOSVQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.455584772 (359h30m45.917677255s) [noise=true , address=45.91.116.6:4973 , piece_id=GW7S6KTE5DLEUMUAEJE27VOFZY5YSTLOPJRCMKRZJCO3WCTQIJEQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.526039842 (359h30m45.988132325s) [noise=true , address=77.90.31.10:4519 , piece_id=GVBAGJMLRINBGUGLR7K5U7U5UDAUO2TDHVQU3YOGH4LUMOZUJP4A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.533295910 (359h30m45.995388403s) [noise=true , address=77.90.31.19:4842 , piece_id=BTCUIS4IDUF5M4JA4HU6NIE3Q5LRX7ORPAM6XB534VPPARXLT7LQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.542682743 (359h30m46.004775216s) [noise=true , address=45.91.116.14:4191 , piece_id=OB5O5DHWAVFLR64FIY2JMETOA6YBVHQZ7X623XG37FISJBLWQKKQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.613214860 (359h30m46.075307343s) [noise=true , address=45.91.116.8:4048 , piece_id=F56BMKSENGC2HXMUOATBI3GSNPPCVVCCVEEQEZVEDI7QSTVE2YHQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.820474816 (359h30m46.282567299s) [noise=true , address=45.91.116.6:4975 , piece_id=P5JV6USSWDB335IRLXXCWMJEHYBUDN6KKCYSTJRWDLUEJEOY7RRA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.831238041 (359h30m46.293330524s) [noise=true , address=45.91.116.7:4009 , piece_id=HUHKWUSQB5LGFWPQZHHS6NTISCESL4WCZ5E5XJCDHK55FBWB6XXA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.849605917 (359h30m46.3116984s) [noise=true , address=45.91.116.12:4225 , piece_id=6WTRWQ625W6WUV2DS27PNLPEWPPSR52H3YFOT6SCMG6TXP6W3J2A, seg_pos={0 0}, upload=316311]: Done uploading piece. err: <nil> Oct 8 13:02:00.849649890 (359h30m46.311742373s) [seg_pos={0 0}, upload=316311]: Segment reached optimal threshold of 13 pieces. Oct 8 13:02:00.849835444 (359h30m46.311927917s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44096->77.90.31.17:4771: use of closed network connection Oct 8 13:02:00.849844051 (359h30m46.311936524s) [noise=true , address=77.90.31.17:4771 , piece_id=3PXQBNPKNSPGWX457DDSCF7GTKENAKTPLOPHE22LLNFBT5SAZMRQ, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849879067 (359h30m46.31197154s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: expected piece hash; context canceled; EOF Oct 8 13:02:00.849888184 (359h30m46.311980657s) [noise=true , address=45.91.116.4:4938 , piece_id=IAHBMRVV72YARM5PTCTUK3LBNLPH6ME6EZQ43EW76UBCEQK5N5RA, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849927940 (359h30m46.312020413s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Oct 8 13:02:00.849934363 (359h30m46.312026836s) [noise=true , address=45.91.116.8:4045 , piece_id=RH3GWGTRTNKP2BZWOPUQSB75VRK63CPWKYJBGQ5PUYACKF46BGRQ, seg_pos={0 0}, upload=316311]: Long tail cancellation due to optimal threshold Oct 8 13:02:00.849955342 (359h30m46.312047825s) [seg_pos={0 0}, upload=316311]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 8 13:02:00.850025265 (359h30m46.312117748s) [upload=316311]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 8 13:02:00.850030355 (359h30m46.312122828s) [upload=316311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 8 13:02:01.159121227 (359h30m46.62121371s) [upload=316311]: Done uploading segments... Oct 8 13:02:01.159145763 (359h30m46.621238246s) [upload=316311]: Upload finished. err: <nil> Duration: 9.222461786s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.162077127s FAIL (2025-10-08 13:10:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.162106904s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.25725126s FAIL (2025-10-08 13:10:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.257275217s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.305902602s FAIL (2025-10-08 13:10:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.30595333s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)