Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.................X.....................X.X.............] duration: 6.240180196s FAIL (2025-09-07 17:07:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:56:07.204847944 (27h27m43.244421064s) [upload=24175]: Starting upload Sep 7 16:56:07.204911426 (27h27m43.244484546s) [upload=24175]: Done starting upload Sep 7 16:56:07.205036886 (27h27m43.244609996s) [upload=24175]: Uploading segments... Sep 7 16:56:07.205165652 (27h27m43.244738772s) [upload=24175]: Got next segment. Inline: false Sep 7 16:56:07.205215979 (27h27m43.244789089s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:56:07.205220527 (27h27m43.244793637s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:56:07.464197848 (27h27m43.503770968s) [seg_pos={0 0}, upload=24175]: Begin upload segment... Sep 7 16:56:07.464245869 (27h27m43.503818989s) [seg_pos={0 0}, upload=24175]: Done begin upload segment. Sep 7 16:56:07.464275365 (27h27m43.503848485s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464296005 (27h27m43.503869125s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464684248 (27h27m43.504257368s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464975956 (27h27m43.504549076s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465248017 (27h27m43.504821137s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465322069 (27h27m43.504895189s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465938488 (27h27m43.505511618s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.466391655 (27h27m43.505964775s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467021020 (27h27m43.50659414s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467496980 (27h27m43.5070701s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467839175 (27h27m43.507412305s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468101657 (27h27m43.507674797s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468353089 (27h27m43.507926209s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468567870 (27h27m43.50814099s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468777811 (27h27m43.508350931s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.469009535 (27h27m43.508582655s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:09.248037394 (27h27m45.287610514s) [upload=24175]: Next returned nil segment Sep 7 16:56:09.248068053 (27h27m45.287641173s) [upload=24175]: Waiting for error group managing segments... Sep 7 16:56:09.623994879 (27h27m45.663568s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.655224468 (27h27m45.694797588s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.719617045 (27h27m45.759190165s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.738395548 (27h27m45.777968658s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.745496015 (27h27m45.785069135s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.746979413 (27h27m45.786552533s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.769784752 (27h27m45.809357872s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.774834194 (27h27m45.814407315s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.812449353 (27h27m45.852022473s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.823629496 (27h27m45.863202616s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.830731396 (27h27m45.870304516s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.862510937 (27h27m45.902084057s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960708310 (27h27m46.00028143s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960737767 (27h27m46.000310887s) [seg_pos={0 0}, upload=24175]: Segment reached optimal threshold of 13 pieces. Sep 7 16:56:09.961123174 (27h27m46.000696294s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 7 16:56:09.961156488 (27h27m46.000729608s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection Sep 7 16:56:09.961187256 (27h27m46.000760376s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection Sep 7 16:56:09.961197355 (27h27m46.000770475s) [seg_pos={0 0}, upload=24175]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:56:09.961274373 (27h27m46.000847493s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:56:09.961278250 (27h27m46.00085137s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:56:10.357307474 (27h27m46.396880594s) [upload=24175]: Done uploading segments... Sep 7 16:56:10.357326480 (27h27m46.3968996s) [upload=24175]: Upload finished. err: <nil> Duration: 6.240205725s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.................X.....................X.X............] duration: 6.420520264s FAIL (2025-09-07 17:07:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:56:07.204847944 (27h27m43.244421064s) [upload=24175]: Starting upload Sep 7 16:56:07.204911426 (27h27m43.244484546s) [upload=24175]: Done starting upload Sep 7 16:56:07.205036886 (27h27m43.244609996s) [upload=24175]: Uploading segments... Sep 7 16:56:07.205165652 (27h27m43.244738772s) [upload=24175]: Got next segment. Inline: false Sep 7 16:56:07.205215979 (27h27m43.244789089s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:56:07.205220527 (27h27m43.244793637s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:56:07.464197848 (27h27m43.503770968s) [seg_pos={0 0}, upload=24175]: Begin upload segment... Sep 7 16:56:07.464245869 (27h27m43.503818989s) [seg_pos={0 0}, upload=24175]: Done begin upload segment. Sep 7 16:56:07.464275365 (27h27m43.503848485s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464296005 (27h27m43.503869125s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464684248 (27h27m43.504257368s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464975956 (27h27m43.504549076s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465248017 (27h27m43.504821137s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465322069 (27h27m43.504895189s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465938488 (27h27m43.505511618s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.466391655 (27h27m43.505964775s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467021020 (27h27m43.50659414s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467496980 (27h27m43.5070701s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467839175 (27h27m43.507412305s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468101657 (27h27m43.507674797s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468353089 (27h27m43.507926209s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468567870 (27h27m43.50814099s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468777811 (27h27m43.508350931s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.469009535 (27h27m43.508582655s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:09.248037394 (27h27m45.287610514s) [upload=24175]: Next returned nil segment Sep 7 16:56:09.248068053 (27h27m45.287641173s) [upload=24175]: Waiting for error group managing segments... Sep 7 16:56:09.623994879 (27h27m45.663568s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.655224468 (27h27m45.694797588s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.719617045 (27h27m45.759190165s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.738395548 (27h27m45.777968658s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.745496015 (27h27m45.785069135s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.746979413 (27h27m45.786552533s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.769784752 (27h27m45.809357872s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.774834194 (27h27m45.814407315s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.812449353 (27h27m45.852022473s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.823629496 (27h27m45.863202616s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.830731396 (27h27m45.870304516s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.862510937 (27h27m45.902084057s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960708310 (27h27m46.00028143s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960737767 (27h27m46.000310887s) [seg_pos={0 0}, upload=24175]: Segment reached optimal threshold of 13 pieces. Sep 7 16:56:09.961123174 (27h27m46.000696294s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 7 16:56:09.961156488 (27h27m46.000729608s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection Sep 7 16:56:09.961187256 (27h27m46.000760376s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection Sep 7 16:56:09.961197355 (27h27m46.000770475s) [seg_pos={0 0}, upload=24175]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:56:09.961274373 (27h27m46.000847493s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:56:09.961278250 (27h27m46.00085137s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:56:10.357307474 (27h27m46.396880594s) [upload=24175]: Done uploading segments... Sep 7 16:56:10.357326480 (27h27m46.3968996s) [upload=24175]: Upload finished. err: <nil> Duration: 6.420559749s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.....X.................X.....................X.X...........] duration: 8.230786094s FAIL (2025-09-07 17:06:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:56:07.204847944 (27h27m43.244421064s) [upload=24175]: Starting upload Sep 7 16:56:07.204911426 (27h27m43.244484546s) [upload=24175]: Done starting upload Sep 7 16:56:07.205036886 (27h27m43.244609996s) [upload=24175]: Uploading segments... Sep 7 16:56:07.205165652 (27h27m43.244738772s) [upload=24175]: Got next segment. Inline: false Sep 7 16:56:07.205215979 (27h27m43.244789089s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:56:07.205220527 (27h27m43.244793637s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:56:07.464197848 (27h27m43.503770968s) [seg_pos={0 0}, upload=24175]: Begin upload segment... Sep 7 16:56:07.464245869 (27h27m43.503818989s) [seg_pos={0 0}, upload=24175]: Done begin upload segment. Sep 7 16:56:07.464275365 (27h27m43.503848485s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464296005 (27h27m43.503869125s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464684248 (27h27m43.504257368s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464975956 (27h27m43.504549076s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465248017 (27h27m43.504821137s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465322069 (27h27m43.504895189s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465938488 (27h27m43.505511618s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.466391655 (27h27m43.505964775s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467021020 (27h27m43.50659414s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467496980 (27h27m43.5070701s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467839175 (27h27m43.507412305s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468101657 (27h27m43.507674797s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468353089 (27h27m43.507926209s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468567870 (27h27m43.50814099s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468777811 (27h27m43.508350931s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.469009535 (27h27m43.508582655s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:09.248037394 (27h27m45.287610514s) [upload=24175]: Next returned nil segment Sep 7 16:56:09.248068053 (27h27m45.287641173s) [upload=24175]: Waiting for error group managing segments... Sep 7 16:56:09.623994879 (27h27m45.663568s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.655224468 (27h27m45.694797588s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.719617045 (27h27m45.759190165s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.738395548 (27h27m45.777968658s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.745496015 (27h27m45.785069135s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.746979413 (27h27m45.786552533s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.769784752 (27h27m45.809357872s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.774834194 (27h27m45.814407315s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.812449353 (27h27m45.852022473s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.823629496 (27h27m45.863202616s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.830731396 (27h27m45.870304516s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.862510937 (27h27m45.902084057s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960708310 (27h27m46.00028143s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960737767 (27h27m46.000310887s) [seg_pos={0 0}, upload=24175]: Segment reached optimal threshold of 13 pieces. Sep 7 16:56:09.961123174 (27h27m46.000696294s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 7 16:56:09.961156488 (27h27m46.000729608s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection Sep 7 16:56:09.961187256 (27h27m46.000760376s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection Sep 7 16:56:09.961197355 (27h27m46.000770475s) [seg_pos={0 0}, upload=24175]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:56:09.961274373 (27h27m46.000847493s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:56:09.961278250 (27h27m46.00085137s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:56:10.357307474 (27h27m46.396880594s) [upload=24175]: Done uploading segments... Sep 7 16:56:10.357326480 (27h27m46.3968996s) [upload=24175]: Upload finished. err: <nil> Duration: 8.230812004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.....X.................X.....................X.X..........] duration: 6.648200617s FAIL (2025-09-07 17:05:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:56:07.204847944 (27h27m43.244421064s) [upload=24175]: Starting upload Sep 7 16:56:07.204911426 (27h27m43.244484546s) [upload=24175]: Done starting upload Sep 7 16:56:07.205036886 (27h27m43.244609996s) [upload=24175]: Uploading segments... Sep 7 16:56:07.205165652 (27h27m43.244738772s) [upload=24175]: Got next segment. Inline: false Sep 7 16:56:07.205215979 (27h27m43.244789089s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:56:07.205220527 (27h27m43.244793637s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:56:07.464197848 (27h27m43.503770968s) [seg_pos={0 0}, upload=24175]: Begin upload segment... Sep 7 16:56:07.464245869 (27h27m43.503818989s) [seg_pos={0 0}, upload=24175]: Done begin upload segment. Sep 7 16:56:07.464275365 (27h27m43.503848485s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464296005 (27h27m43.503869125s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464684248 (27h27m43.504257368s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464975956 (27h27m43.504549076s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465248017 (27h27m43.504821137s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465322069 (27h27m43.504895189s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465938488 (27h27m43.505511618s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.466391655 (27h27m43.505964775s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467021020 (27h27m43.50659414s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467496980 (27h27m43.5070701s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467839175 (27h27m43.507412305s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468101657 (27h27m43.507674797s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468353089 (27h27m43.507926209s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468567870 (27h27m43.50814099s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468777811 (27h27m43.508350931s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.469009535 (27h27m43.508582655s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:09.248037394 (27h27m45.287610514s) [upload=24175]: Next returned nil segment Sep 7 16:56:09.248068053 (27h27m45.287641173s) [upload=24175]: Waiting for error group managing segments... Sep 7 16:56:09.623994879 (27h27m45.663568s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.655224468 (27h27m45.694797588s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.719617045 (27h27m45.759190165s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.738395548 (27h27m45.777968658s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.745496015 (27h27m45.785069135s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.746979413 (27h27m45.786552533s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.769784752 (27h27m45.809357872s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.774834194 (27h27m45.814407315s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.812449353 (27h27m45.852022473s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.823629496 (27h27m45.863202616s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.830731396 (27h27m45.870304516s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.862510937 (27h27m45.902084057s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960708310 (27h27m46.00028143s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960737767 (27h27m46.000310887s) [seg_pos={0 0}, upload=24175]: Segment reached optimal threshold of 13 pieces. Sep 7 16:56:09.961123174 (27h27m46.000696294s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 7 16:56:09.961156488 (27h27m46.000729608s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection Sep 7 16:56:09.961187256 (27h27m46.000760376s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection Sep 7 16:56:09.961197355 (27h27m46.000770475s) [seg_pos={0 0}, upload=24175]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:56:09.961274373 (27h27m46.000847493s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:56:09.961278250 (27h27m46.00085137s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:56:10.357307474 (27h27m46.396880594s) [upload=24175]: Done uploading segments... Sep 7 16:56:10.357326480 (27h27m46.3968996s) [upload=24175]: Upload finished. err: <nil> Duration: 6.648232639s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.....X.................X.....................X.X.........] duration: 23.166268064s FAIL (2025-09-07 17:04:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:56:07.204847944 (27h27m43.244421064s) [upload=24175]: Starting upload Sep 7 16:56:07.204911426 (27h27m43.244484546s) [upload=24175]: Done starting upload Sep 7 16:56:07.205036886 (27h27m43.244609996s) [upload=24175]: Uploading segments... Sep 7 16:56:07.205165652 (27h27m43.244738772s) [upload=24175]: Got next segment. Inline: false Sep 7 16:56:07.205215979 (27h27m43.244789089s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:56:07.205220527 (27h27m43.244793637s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:56:07.464197848 (27h27m43.503770968s) [seg_pos={0 0}, upload=24175]: Begin upload segment... Sep 7 16:56:07.464245869 (27h27m43.503818989s) [seg_pos={0 0}, upload=24175]: Done begin upload segment. Sep 7 16:56:07.464275365 (27h27m43.503848485s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464296005 (27h27m43.503869125s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464684248 (27h27m43.504257368s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464975956 (27h27m43.504549076s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465248017 (27h27m43.504821137s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465322069 (27h27m43.504895189s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465938488 (27h27m43.505511618s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.466391655 (27h27m43.505964775s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467021020 (27h27m43.50659414s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467496980 (27h27m43.5070701s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467839175 (27h27m43.507412305s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468101657 (27h27m43.507674797s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468353089 (27h27m43.507926209s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468567870 (27h27m43.50814099s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468777811 (27h27m43.508350931s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.469009535 (27h27m43.508582655s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:09.248037394 (27h27m45.287610514s) [upload=24175]: Next returned nil segment Sep 7 16:56:09.248068053 (27h27m45.287641173s) [upload=24175]: Waiting for error group managing segments... Sep 7 16:56:09.623994879 (27h27m45.663568s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.655224468 (27h27m45.694797588s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.719617045 (27h27m45.759190165s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.738395548 (27h27m45.777968658s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.745496015 (27h27m45.785069135s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.746979413 (27h27m45.786552533s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.769784752 (27h27m45.809357872s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.774834194 (27h27m45.814407315s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.812449353 (27h27m45.852022473s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.823629496 (27h27m45.863202616s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.830731396 (27h27m45.870304516s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.862510937 (27h27m45.902084057s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960708310 (27h27m46.00028143s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960737767 (27h27m46.000310887s) [seg_pos={0 0}, upload=24175]: Segment reached optimal threshold of 13 pieces. Sep 7 16:56:09.961123174 (27h27m46.000696294s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 7 16:56:09.961156488 (27h27m46.000729608s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection Sep 7 16:56:09.961187256 (27h27m46.000760376s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection Sep 7 16:56:09.961197355 (27h27m46.000770475s) [seg_pos={0 0}, upload=24175]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:56:09.961274373 (27h27m46.000847493s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:56:09.961278250 (27h27m46.00085137s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:56:10.357307474 (27h27m46.396880594s) [upload=24175]: Done uploading segments... Sep 7 16:56:10.357326480 (27h27m46.3968996s) [upload=24175]: Upload finished. err: <nil> Duration: 23.166291978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.....X.................X.....................X.X........] duration: 11.568923748s FAIL (2025-09-07 17:03:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:56:07.204847944 (27h27m43.244421064s) [upload=24175]: Starting upload Sep 7 16:56:07.204911426 (27h27m43.244484546s) [upload=24175]: Done starting upload Sep 7 16:56:07.205036886 (27h27m43.244609996s) [upload=24175]: Uploading segments... Sep 7 16:56:07.205165652 (27h27m43.244738772s) [upload=24175]: Got next segment. Inline: false Sep 7 16:56:07.205215979 (27h27m43.244789089s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:56:07.205220527 (27h27m43.244793637s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:56:07.464197848 (27h27m43.503770968s) [seg_pos={0 0}, upload=24175]: Begin upload segment... Sep 7 16:56:07.464245869 (27h27m43.503818989s) [seg_pos={0 0}, upload=24175]: Done begin upload segment. Sep 7 16:56:07.464275365 (27h27m43.503848485s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464296005 (27h27m43.503869125s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464684248 (27h27m43.504257368s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464975956 (27h27m43.504549076s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465248017 (27h27m43.504821137s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465322069 (27h27m43.504895189s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465938488 (27h27m43.505511618s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.466391655 (27h27m43.505964775s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467021020 (27h27m43.50659414s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467496980 (27h27m43.5070701s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467839175 (27h27m43.507412305s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468101657 (27h27m43.507674797s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468353089 (27h27m43.507926209s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468567870 (27h27m43.50814099s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468777811 (27h27m43.508350931s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.469009535 (27h27m43.508582655s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:09.248037394 (27h27m45.287610514s) [upload=24175]: Next returned nil segment Sep 7 16:56:09.248068053 (27h27m45.287641173s) [upload=24175]: Waiting for error group managing segments... Sep 7 16:56:09.623994879 (27h27m45.663568s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.655224468 (27h27m45.694797588s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.719617045 (27h27m45.759190165s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.738395548 (27h27m45.777968658s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.745496015 (27h27m45.785069135s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.746979413 (27h27m45.786552533s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.769784752 (27h27m45.809357872s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.774834194 (27h27m45.814407315s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.812449353 (27h27m45.852022473s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.823629496 (27h27m45.863202616s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.830731396 (27h27m45.870304516s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.862510937 (27h27m45.902084057s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960708310 (27h27m46.00028143s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960737767 (27h27m46.000310887s) [seg_pos={0 0}, upload=24175]: Segment reached optimal threshold of 13 pieces. Sep 7 16:56:09.961123174 (27h27m46.000696294s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 7 16:56:09.961156488 (27h27m46.000729608s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection Sep 7 16:56:09.961187256 (27h27m46.000760376s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection Sep 7 16:56:09.961197355 (27h27m46.000770475s) [seg_pos={0 0}, upload=24175]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:56:09.961274373 (27h27m46.000847493s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:56:09.961278250 (27h27m46.00085137s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:56:10.357307474 (27h27m46.396880594s) [upload=24175]: Done uploading segments... Sep 7 16:56:10.357326480 (27h27m46.3968996s) [upload=24175]: Upload finished. err: <nil> Duration: 11.568947955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.....X.................X.....................X.X.......] duration: 9.562026099s FAIL (2025-09-07 17:02:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:56:07.204847944 (27h27m43.244421064s) [upload=24175]: Starting upload Sep 7 16:56:07.204911426 (27h27m43.244484546s) [upload=24175]: Done starting upload Sep 7 16:56:07.205036886 (27h27m43.244609996s) [upload=24175]: Uploading segments... Sep 7 16:56:07.205165652 (27h27m43.244738772s) [upload=24175]: Got next segment. Inline: false Sep 7 16:56:07.205215979 (27h27m43.244789089s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:56:07.205220527 (27h27m43.244793637s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:56:07.464197848 (27h27m43.503770968s) [seg_pos={0 0}, upload=24175]: Begin upload segment... Sep 7 16:56:07.464245869 (27h27m43.503818989s) [seg_pos={0 0}, upload=24175]: Done begin upload segment. Sep 7 16:56:07.464275365 (27h27m43.503848485s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464296005 (27h27m43.503869125s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464684248 (27h27m43.504257368s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464975956 (27h27m43.504549076s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465248017 (27h27m43.504821137s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465322069 (27h27m43.504895189s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465938488 (27h27m43.505511618s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.466391655 (27h27m43.505964775s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467021020 (27h27m43.50659414s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467496980 (27h27m43.5070701s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467839175 (27h27m43.507412305s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468101657 (27h27m43.507674797s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468353089 (27h27m43.507926209s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468567870 (27h27m43.50814099s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468777811 (27h27m43.508350931s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.469009535 (27h27m43.508582655s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:09.248037394 (27h27m45.287610514s) [upload=24175]: Next returned nil segment Sep 7 16:56:09.248068053 (27h27m45.287641173s) [upload=24175]: Waiting for error group managing segments... Sep 7 16:56:09.623994879 (27h27m45.663568s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.655224468 (27h27m45.694797588s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.719617045 (27h27m45.759190165s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.738395548 (27h27m45.777968658s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.745496015 (27h27m45.785069135s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.746979413 (27h27m45.786552533s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.769784752 (27h27m45.809357872s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.774834194 (27h27m45.814407315s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.812449353 (27h27m45.852022473s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.823629496 (27h27m45.863202616s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.830731396 (27h27m45.870304516s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.862510937 (27h27m45.902084057s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960708310 (27h27m46.00028143s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960737767 (27h27m46.000310887s) [seg_pos={0 0}, upload=24175]: Segment reached optimal threshold of 13 pieces. Sep 7 16:56:09.961123174 (27h27m46.000696294s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 7 16:56:09.961156488 (27h27m46.000729608s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection Sep 7 16:56:09.961187256 (27h27m46.000760376s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection Sep 7 16:56:09.961197355 (27h27m46.000770475s) [seg_pos={0 0}, upload=24175]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:56:09.961274373 (27h27m46.000847493s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:56:09.961278250 (27h27m46.00085137s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:56:10.357307474 (27h27m46.396880594s) [upload=24175]: Done uploading segments... Sep 7 16:56:10.357326480 (27h27m46.3968996s) [upload=24175]: Upload finished. err: <nil> Duration: 9.562058872s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.....X.................X.....................X.X......] duration: 24.830415737s FAIL (2025-09-07 17:02:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:56:07.204847944 (27h27m43.244421064s) [upload=24175]: Starting upload Sep 7 16:56:07.204911426 (27h27m43.244484546s) [upload=24175]: Done starting upload Sep 7 16:56:07.205036886 (27h27m43.244609996s) [upload=24175]: Uploading segments... Sep 7 16:56:07.205165652 (27h27m43.244738772s) [upload=24175]: Got next segment. Inline: false Sep 7 16:56:07.205215979 (27h27m43.244789089s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:56:07.205220527 (27h27m43.244793637s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:56:07.464197848 (27h27m43.503770968s) [seg_pos={0 0}, upload=24175]: Begin upload segment... Sep 7 16:56:07.464245869 (27h27m43.503818989s) [seg_pos={0 0}, upload=24175]: Done begin upload segment. Sep 7 16:56:07.464275365 (27h27m43.503848485s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464296005 (27h27m43.503869125s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464684248 (27h27m43.504257368s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464975956 (27h27m43.504549076s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465248017 (27h27m43.504821137s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465322069 (27h27m43.504895189s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465938488 (27h27m43.505511618s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.466391655 (27h27m43.505964775s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467021020 (27h27m43.50659414s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467496980 (27h27m43.5070701s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467839175 (27h27m43.507412305s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468101657 (27h27m43.507674797s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468353089 (27h27m43.507926209s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468567870 (27h27m43.50814099s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468777811 (27h27m43.508350931s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.469009535 (27h27m43.508582655s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:09.248037394 (27h27m45.287610514s) [upload=24175]: Next returned nil segment Sep 7 16:56:09.248068053 (27h27m45.287641173s) [upload=24175]: Waiting for error group managing segments... Sep 7 16:56:09.623994879 (27h27m45.663568s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.655224468 (27h27m45.694797588s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.719617045 (27h27m45.759190165s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.738395548 (27h27m45.777968658s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.745496015 (27h27m45.785069135s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.746979413 (27h27m45.786552533s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.769784752 (27h27m45.809357872s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.774834194 (27h27m45.814407315s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.812449353 (27h27m45.852022473s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.823629496 (27h27m45.863202616s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.830731396 (27h27m45.870304516s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.862510937 (27h27m45.902084057s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960708310 (27h27m46.00028143s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960737767 (27h27m46.000310887s) [seg_pos={0 0}, upload=24175]: Segment reached optimal threshold of 13 pieces. Sep 7 16:56:09.961123174 (27h27m46.000696294s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 7 16:56:09.961156488 (27h27m46.000729608s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection Sep 7 16:56:09.961187256 (27h27m46.000760376s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection Sep 7 16:56:09.961197355 (27h27m46.000770475s) [seg_pos={0 0}, upload=24175]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:56:09.961274373 (27h27m46.000847493s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:56:09.961278250 (27h27m46.00085137s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:56:10.357307474 (27h27m46.396880594s) [upload=24175]: Done uploading segments... Sep 7 16:56:10.357326480 (27h27m46.3968996s) [upload=24175]: Upload finished. err: <nil> Duration: 24.83044319s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.....X.....X.................X.....................X.X.....] duration: 6.63556538s FAIL (2025-09-07 17:00:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:56:07.204847944 (27h27m43.244421064s) [upload=24175]: Starting upload Sep 7 16:56:07.204911426 (27h27m43.244484546s) [upload=24175]: Done starting upload Sep 7 16:56:07.205036886 (27h27m43.244609996s) [upload=24175]: Uploading segments... Sep 7 16:56:07.205165652 (27h27m43.244738772s) [upload=24175]: Got next segment. Inline: false Sep 7 16:56:07.205215979 (27h27m43.244789089s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:56:07.205220527 (27h27m43.244793637s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:56:07.464197848 (27h27m43.503770968s) [seg_pos={0 0}, upload=24175]: Begin upload segment... Sep 7 16:56:07.464245869 (27h27m43.503818989s) [seg_pos={0 0}, upload=24175]: Done begin upload segment. Sep 7 16:56:07.464275365 (27h27m43.503848485s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464296005 (27h27m43.503869125s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464684248 (27h27m43.504257368s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464975956 (27h27m43.504549076s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465248017 (27h27m43.504821137s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465322069 (27h27m43.504895189s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465938488 (27h27m43.505511618s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.466391655 (27h27m43.505964775s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467021020 (27h27m43.50659414s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467496980 (27h27m43.5070701s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467839175 (27h27m43.507412305s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468101657 (27h27m43.507674797s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468353089 (27h27m43.507926209s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468567870 (27h27m43.50814099s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468777811 (27h27m43.508350931s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.469009535 (27h27m43.508582655s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:09.248037394 (27h27m45.287610514s) [upload=24175]: Next returned nil segment Sep 7 16:56:09.248068053 (27h27m45.287641173s) [upload=24175]: Waiting for error group managing segments... Sep 7 16:56:09.623994879 (27h27m45.663568s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.655224468 (27h27m45.694797588s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.719617045 (27h27m45.759190165s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.738395548 (27h27m45.777968658s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.745496015 (27h27m45.785069135s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.746979413 (27h27m45.786552533s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.769784752 (27h27m45.809357872s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.774834194 (27h27m45.814407315s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.812449353 (27h27m45.852022473s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.823629496 (27h27m45.863202616s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.830731396 (27h27m45.870304516s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.862510937 (27h27m45.902084057s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960708310 (27h27m46.00028143s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960737767 (27h27m46.000310887s) [seg_pos={0 0}, upload=24175]: Segment reached optimal threshold of 13 pieces. Sep 7 16:56:09.961123174 (27h27m46.000696294s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 7 16:56:09.961156488 (27h27m46.000729608s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection Sep 7 16:56:09.961187256 (27h27m46.000760376s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection Sep 7 16:56:09.961197355 (27h27m46.000770475s) [seg_pos={0 0}, upload=24175]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:56:09.961274373 (27h27m46.000847493s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:56:09.961278250 (27h27m46.00085137s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:56:10.357307474 (27h27m46.396880594s) [upload=24175]: Done uploading segments... Sep 7 16:56:10.357326480 (27h27m46.3968996s) [upload=24175]: Upload finished. err: <nil> Duration: 6.635583935s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.....X.....X.................X.....................X.X....] duration: 30.001633364s FAIL (2025-09-07 17:00:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:56:07.204847944 (27h27m43.244421064s) [upload=24175]: Starting upload Sep 7 16:56:07.204911426 (27h27m43.244484546s) [upload=24175]: Done starting upload Sep 7 16:56:07.205036886 (27h27m43.244609996s) [upload=24175]: Uploading segments... Sep 7 16:56:07.205165652 (27h27m43.244738772s) [upload=24175]: Got next segment. Inline: false Sep 7 16:56:07.205215979 (27h27m43.244789089s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:56:07.205220527 (27h27m43.244793637s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:56:07.464197848 (27h27m43.503770968s) [seg_pos={0 0}, upload=24175]: Begin upload segment... Sep 7 16:56:07.464245869 (27h27m43.503818989s) [seg_pos={0 0}, upload=24175]: Done begin upload segment. Sep 7 16:56:07.464275365 (27h27m43.503848485s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464296005 (27h27m43.503869125s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464684248 (27h27m43.504257368s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464975956 (27h27m43.504549076s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465248017 (27h27m43.504821137s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465322069 (27h27m43.504895189s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465938488 (27h27m43.505511618s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.466391655 (27h27m43.505964775s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467021020 (27h27m43.50659414s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467496980 (27h27m43.5070701s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467839175 (27h27m43.507412305s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468101657 (27h27m43.507674797s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468353089 (27h27m43.507926209s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468567870 (27h27m43.50814099s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468777811 (27h27m43.508350931s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.469009535 (27h27m43.508582655s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:09.248037394 (27h27m45.287610514s) [upload=24175]: Next returned nil segment Sep 7 16:56:09.248068053 (27h27m45.287641173s) [upload=24175]: Waiting for error group managing segments... Sep 7 16:56:09.623994879 (27h27m45.663568s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.655224468 (27h27m45.694797588s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.719617045 (27h27m45.759190165s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.738395548 (27h27m45.777968658s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.745496015 (27h27m45.785069135s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.746979413 (27h27m45.786552533s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.769784752 (27h27m45.809357872s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.774834194 (27h27m45.814407315s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.812449353 (27h27m45.852022473s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.823629496 (27h27m45.863202616s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.830731396 (27h27m45.870304516s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.862510937 (27h27m45.902084057s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960708310 (27h27m46.00028143s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960737767 (27h27m46.000310887s) [seg_pos={0 0}, upload=24175]: Segment reached optimal threshold of 13 pieces. Sep 7 16:56:09.961123174 (27h27m46.000696294s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 7 16:56:09.961156488 (27h27m46.000729608s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection Sep 7 16:56:09.961187256 (27h27m46.000760376s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection Sep 7 16:56:09.961197355 (27h27m46.000770475s) [seg_pos={0 0}, upload=24175]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:56:09.961274373 (27h27m46.000847493s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:56:09.961278250 (27h27m46.00085137s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:56:10.357307474 (27h27m46.396880594s) [upload=24175]: Done uploading segments... Sep 7 16:56:10.357326480 (27h27m46.3968996s) [upload=24175]: Upload finished. err: <nil> Duration: 30.001657961s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.....X.....X.................X.....................X.X...] duration: 6.501984759s FAIL (2025-09-07 16:58:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:56:07.204847944 (27h27m43.244421064s) [upload=24175]: Starting upload Sep 7 16:56:07.204911426 (27h27m43.244484546s) [upload=24175]: Done starting upload Sep 7 16:56:07.205036886 (27h27m43.244609996s) [upload=24175]: Uploading segments... Sep 7 16:56:07.205165652 (27h27m43.244738772s) [upload=24175]: Got next segment. Inline: false Sep 7 16:56:07.205215979 (27h27m43.244789089s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:56:07.205220527 (27h27m43.244793637s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:56:07.464197848 (27h27m43.503770968s) [seg_pos={0 0}, upload=24175]: Begin upload segment... Sep 7 16:56:07.464245869 (27h27m43.503818989s) [seg_pos={0 0}, upload=24175]: Done begin upload segment. Sep 7 16:56:07.464275365 (27h27m43.503848485s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464296005 (27h27m43.503869125s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464684248 (27h27m43.504257368s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464975956 (27h27m43.504549076s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465248017 (27h27m43.504821137s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465322069 (27h27m43.504895189s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465938488 (27h27m43.505511618s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.466391655 (27h27m43.505964775s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467021020 (27h27m43.50659414s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467496980 (27h27m43.5070701s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467839175 (27h27m43.507412305s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468101657 (27h27m43.507674797s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468353089 (27h27m43.507926209s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468567870 (27h27m43.50814099s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468777811 (27h27m43.508350931s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.469009535 (27h27m43.508582655s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:09.248037394 (27h27m45.287610514s) [upload=24175]: Next returned nil segment Sep 7 16:56:09.248068053 (27h27m45.287641173s) [upload=24175]: Waiting for error group managing segments... Sep 7 16:56:09.623994879 (27h27m45.663568s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.655224468 (27h27m45.694797588s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.719617045 (27h27m45.759190165s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.738395548 (27h27m45.777968658s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.745496015 (27h27m45.785069135s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.746979413 (27h27m45.786552533s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.769784752 (27h27m45.809357872s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.774834194 (27h27m45.814407315s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.812449353 (27h27m45.852022473s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.823629496 (27h27m45.863202616s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.830731396 (27h27m45.870304516s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.862510937 (27h27m45.902084057s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960708310 (27h27m46.00028143s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960737767 (27h27m46.000310887s) [seg_pos={0 0}, upload=24175]: Segment reached optimal threshold of 13 pieces. Sep 7 16:56:09.961123174 (27h27m46.000696294s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 7 16:56:09.961156488 (27h27m46.000729608s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection Sep 7 16:56:09.961187256 (27h27m46.000760376s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection Sep 7 16:56:09.961197355 (27h27m46.000770475s) [seg_pos={0 0}, upload=24175]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:56:09.961274373 (27h27m46.000847493s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:56:09.961278250 (27h27m46.00085137s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:56:10.357307474 (27h27m46.396880594s) [upload=24175]: Done uploading segments... Sep 7 16:56:10.357326480 (27h27m46.3968996s) [upload=24175]: Upload finished. err: <nil> Duration: 6.50201197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.....X.....X.................X.....................X.X..] duration: 6.525000228s FAIL (2025-09-07 16:57:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:56:07.204847944 (27h27m43.244421064s) [upload=24175]: Starting upload Sep 7 16:56:07.204911426 (27h27m43.244484546s) [upload=24175]: Done starting upload Sep 7 16:56:07.205036886 (27h27m43.244609996s) [upload=24175]: Uploading segments... Sep 7 16:56:07.205165652 (27h27m43.244738772s) [upload=24175]: Got next segment. Inline: false Sep 7 16:56:07.205215979 (27h27m43.244789089s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:56:07.205220527 (27h27m43.244793637s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:56:07.464197848 (27h27m43.503770968s) [seg_pos={0 0}, upload=24175]: Begin upload segment... Sep 7 16:56:07.464245869 (27h27m43.503818989s) [seg_pos={0 0}, upload=24175]: Done begin upload segment. Sep 7 16:56:07.464275365 (27h27m43.503848485s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464296005 (27h27m43.503869125s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464684248 (27h27m43.504257368s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464975956 (27h27m43.504549076s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465248017 (27h27m43.504821137s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465322069 (27h27m43.504895189s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465938488 (27h27m43.505511618s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.466391655 (27h27m43.505964775s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467021020 (27h27m43.50659414s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467496980 (27h27m43.5070701s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467839175 (27h27m43.507412305s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468101657 (27h27m43.507674797s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468353089 (27h27m43.507926209s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468567870 (27h27m43.50814099s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468777811 (27h27m43.508350931s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.469009535 (27h27m43.508582655s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:09.248037394 (27h27m45.287610514s) [upload=24175]: Next returned nil segment Sep 7 16:56:09.248068053 (27h27m45.287641173s) [upload=24175]: Waiting for error group managing segments... Sep 7 16:56:09.623994879 (27h27m45.663568s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.655224468 (27h27m45.694797588s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.719617045 (27h27m45.759190165s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.738395548 (27h27m45.777968658s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.745496015 (27h27m45.785069135s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.746979413 (27h27m45.786552533s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.769784752 (27h27m45.809357872s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.774834194 (27h27m45.814407315s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.812449353 (27h27m45.852022473s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.823629496 (27h27m45.863202616s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.830731396 (27h27m45.870304516s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.862510937 (27h27m45.902084057s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960708310 (27h27m46.00028143s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960737767 (27h27m46.000310887s) [seg_pos={0 0}, upload=24175]: Segment reached optimal threshold of 13 pieces. Sep 7 16:56:09.961123174 (27h27m46.000696294s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 7 16:56:09.961156488 (27h27m46.000729608s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection Sep 7 16:56:09.961187256 (27h27m46.000760376s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection Sep 7 16:56:09.961197355 (27h27m46.000770475s) [seg_pos={0 0}, upload=24175]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:56:09.961274373 (27h27m46.000847493s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:56:09.961278250 (27h27m46.00085137s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:56:10.357307474 (27h27m46.396880594s) [upload=24175]: Done uploading segments... Sep 7 16:56:10.357326480 (27h27m46.3968996s) [upload=24175]: Upload finished. err: <nil> Duration: 6.52502761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X.....X.....X.................X.....................X.X.] duration: 6.30059834s FAIL (2025-09-07 16:56:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:56:07.204847944 (27h27m43.244421064s) [upload=24175]: Starting upload Sep 7 16:56:07.204911426 (27h27m43.244484546s) [upload=24175]: Done starting upload Sep 7 16:56:07.205036886 (27h27m43.244609996s) [upload=24175]: Uploading segments... Sep 7 16:56:07.205165652 (27h27m43.244738772s) [upload=24175]: Got next segment. Inline: false Sep 7 16:56:07.205215979 (27h27m43.244789089s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:56:07.205220527 (27h27m43.244793637s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:56:07.464197848 (27h27m43.503770968s) [seg_pos={0 0}, upload=24175]: Begin upload segment... Sep 7 16:56:07.464245869 (27h27m43.503818989s) [seg_pos={0 0}, upload=24175]: Done begin upload segment. Sep 7 16:56:07.464275365 (27h27m43.503848485s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464296005 (27h27m43.503869125s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464684248 (27h27m43.504257368s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464975956 (27h27m43.504549076s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465248017 (27h27m43.504821137s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465322069 (27h27m43.504895189s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465938488 (27h27m43.505511618s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.466391655 (27h27m43.505964775s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467021020 (27h27m43.50659414s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467496980 (27h27m43.5070701s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467839175 (27h27m43.507412305s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468101657 (27h27m43.507674797s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468353089 (27h27m43.507926209s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468567870 (27h27m43.50814099s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468777811 (27h27m43.508350931s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.469009535 (27h27m43.508582655s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:09.248037394 (27h27m45.287610514s) [upload=24175]: Next returned nil segment Sep 7 16:56:09.248068053 (27h27m45.287641173s) [upload=24175]: Waiting for error group managing segments... Sep 7 16:56:09.623994879 (27h27m45.663568s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.655224468 (27h27m45.694797588s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.719617045 (27h27m45.759190165s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.738395548 (27h27m45.777968658s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.745496015 (27h27m45.785069135s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.746979413 (27h27m45.786552533s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.769784752 (27h27m45.809357872s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.774834194 (27h27m45.814407315s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.812449353 (27h27m45.852022473s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.823629496 (27h27m45.863202616s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.830731396 (27h27m45.870304516s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.862510937 (27h27m45.902084057s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960708310 (27h27m46.00028143s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960737767 (27h27m46.000310887s) [seg_pos={0 0}, upload=24175]: Segment reached optimal threshold of 13 pieces. Sep 7 16:56:09.961123174 (27h27m46.000696294s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 7 16:56:09.961156488 (27h27m46.000729608s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection Sep 7 16:56:09.961187256 (27h27m46.000760376s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection Sep 7 16:56:09.961197355 (27h27m46.000770475s) [seg_pos={0 0}, upload=24175]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:56:09.961274373 (27h27m46.000847493s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:56:09.961278250 (27h27m46.00085137s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:56:10.357307474 (27h27m46.396880594s) [upload=24175]: Done uploading segments... Sep 7 16:56:10.357326480 (27h27m46.3968996s) [upload=24175]: Upload finished. err: <nil> Duration: 6.300628026s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X.....X.....X.................X.....................X.X] duration: 30.001979352s FAIL (2025-09-07 16:56:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:56:07.204847944 (27h27m43.244421064s) [upload=24175]: Starting upload Sep 7 16:56:07.204911426 (27h27m43.244484546s) [upload=24175]: Done starting upload Sep 7 16:56:07.205036886 (27h27m43.244609996s) [upload=24175]: Uploading segments... Sep 7 16:56:07.205165652 (27h27m43.244738772s) [upload=24175]: Got next segment. Inline: false Sep 7 16:56:07.205215979 (27h27m43.244789089s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:56:07.205220527 (27h27m43.244793637s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:56:07.464197848 (27h27m43.503770968s) [seg_pos={0 0}, upload=24175]: Begin upload segment... Sep 7 16:56:07.464245869 (27h27m43.503818989s) [seg_pos={0 0}, upload=24175]: Done begin upload segment. Sep 7 16:56:07.464275365 (27h27m43.503848485s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464296005 (27h27m43.503869125s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464684248 (27h27m43.504257368s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.464975956 (27h27m43.504549076s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465248017 (27h27m43.504821137s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465322069 (27h27m43.504895189s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.465938488 (27h27m43.505511618s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.466391655 (27h27m43.505964775s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467021020 (27h27m43.50659414s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467496980 (27h27m43.5070701s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.467839175 (27h27m43.507412305s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468101657 (27h27m43.507674797s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468353089 (27h27m43.507926209s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468567870 (27h27m43.50814099s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.468777811 (27h27m43.508350931s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:07.469009535 (27h27m43.508582655s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Uploading piece... Sep 7 16:56:09.248037394 (27h27m45.287610514s) [upload=24175]: Next returned nil segment Sep 7 16:56:09.248068053 (27h27m45.287641173s) [upload=24175]: Waiting for error group managing segments... Sep 7 16:56:09.623994879 (27h27m45.663568s) [noise=true , address=45.91.116.6:4974 , piece_id=JJJUEUV5W5ZUO3SZEWPHDX5I3E2XMEUIM5X4PYC4P7NTP25ZFNYQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.655224468 (27h27m45.694797588s) [noise=true , address=45.91.116.6:4975 , piece_id=ZAGONBY6YHH2R4STFBCWCEZ35OOR6XKDZN23U7MUDDQMJ22LN6QA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.719617045 (27h27m45.759190165s) [noise=true , address=77.90.31.10:4519 , piece_id=Q5UT73AWWUK5ER7I7ZS6SX5KEI4QRZ6RHMXX74SZA3H3FWJV4NKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.738395548 (27h27m45.777968658s) [noise=true , address=45.91.116.4:4937 , piece_id=XR5EWBR66M2PITA7MHWDMLBPXDOFALTEYMP6RANRE6DWWZSJNM6A, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.745496015 (27h27m45.785069135s) [noise=true , address=77.90.31.10:4518 , piece_id=46VRAZ5DH5PYII5PBATPC5OMA4PQTAIWSN5AO7IMSIG6K3ULR2UQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.746979413 (27h27m45.786552533s) [noise=true , address=45.91.116.10:4081 , piece_id=6UMDYW4IJRPYWRL4FYFHT3BCQH5G3ZBBK37OZFBEI3ADBSLX7VWQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.769784752 (27h27m45.809357872s) [noise=true , address=77.90.31.15:4698 , piece_id=JGQZK7LWPPMSGU3R3FHRQQH7ONFCYNUIJXK3CT6FYGKIJZHBOFLQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.774834194 (27h27m45.814407315s) [noise=true , address=77.90.31.16:4734 , piece_id=W535J6GJGUM5DJWJXPTIWRZ6FNZWILATT5PUNMLDUJVR2IGTZFJQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.812449353 (27h27m45.852022473s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTQU4EUXJEJJ6YMQDCFMXKXQ6PEASUSQDI342BNT7VHDKS6OUFFA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.823629496 (27h27m45.863202616s) [noise=true , address=45.91.116.5:4297 , piece_id=UPLJMBWDGTV7XALLGDHKTZO5WB5USC2AZKEYBIHCUIF4OB4NEKGA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.830731396 (27h27m45.870304516s) [noise=true , address=77.90.31.12:4590 , piece_id=HQUJ626TG6G3K7HBM3TWAWWO5U5T6AYXWIYSSKKNN4OLXL3P45BQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.862510937 (27h27m45.902084057s) [noise=true , address=45.91.116.7:4009 , piece_id=4BVACPLFNR6HYYP3ND5XD4TDQTMCJZZBZGC4WXXSPW5NMZEKIIQA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960708310 (27h27m46.00028143s) [noise=true , address=77.90.31.18:4807 , piece_id=JJOXCFFDNB3FUGNAXQXO33WFXQOKPBHP3CNGW52C23TW5QBGD52Q, seg_pos={0 0}, upload=24175]: Done uploading piece. err: <nil> Sep 7 16:56:09.960737767 (27h27m46.000310887s) [seg_pos={0 0}, upload=24175]: Segment reached optimal threshold of 13 pieces. Sep 7 16:56:09.961123174 (27h27m46.000696294s) [noise=true , address=77.90.31.19:4842 , piece_id=3LSSUTIKRRVJXPVPHTP6GHE55XHSZTTFSWXDKBPAQCKXTAGZDYLA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 7 16:56:09.961156488 (27h27m46.000729608s) [noise=true , address=45.91.116.8:4048 , piece_id=V5CEPRKF5B7KCQIY4HMAQPTM22ABRDG2SV5KFO4GKDRGJCDJ2RKQ, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60946->45.91.116.8:4048: use of closed network connection Sep 7 16:56:09.961187256 (27h27m46.000760376s) [noise=true , address=45.91.116.4:4939 , piece_id=LV7KI4YZFFSJBMVTNIXQAH2GL2XYXTOFSRKLGQ3LIELOMEMELOIA, seg_pos={0 0}, upload=24175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47690->45.91.116.4:4939: use of closed network connection Sep 7 16:56:09.961197355 (27h27m46.000770475s) [seg_pos={0 0}, upload=24175]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:56:09.961274373 (27h27m46.000847493s) [upload=24175]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:56:09.961278250 (27h27m46.00085137s) [upload=24175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:56:10.357307474 (27h27m46.396880594s) [upload=24175]: Done uploading segments... Sep 7 16:56:10.357326480 (27h27m46.3968996s) [upload=24175]: Upload finished. err: <nil> Duration: 30.002005142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.....X.....X.................X.....................X.] duration: 6.60983356s FAIL (2025-09-07 16:55:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:54:37.607783895 (27h26m13.647357015s) [upload=24152]: Starting upload Sep 7 16:54:37.607839852 (27h26m13.647412972s) [upload=24152]: Done starting upload Sep 7 16:54:37.607926227 (27h26m13.647499347s) [upload=24152]: Uploading segments... Sep 7 16:54:37.608030476 (27h26m13.647603596s) [upload=24152]: Got next segment. Inline: false Sep 7 16:54:37.608061005 (27h26m13.647634125s) [upload=24152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:54:37.608065523 (27h26m13.647638643s) [upload=24152]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:54:37.857213566 (27h26m13.896786686s) [seg_pos={0 0}, upload=24152]: Begin upload segment... Sep 7 16:54:37.857253812 (27h26m13.896826932s) [seg_pos={0 0}, upload=24152]: Done begin upload segment. Sep 7 16:54:37.857273811 (27h26m13.896846921s) [noise=true , address=77.90.31.13:4628 , piece_id=G72N6ULIGJC5USBQJBOAUT4QJ57GRK2PAN7CA32UATGQQQR5RTMQ, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.857341290 (27h26m13.89691442s) [noise=true , address=45.91.116.11:4118 , piece_id=FR3OPJNV6WJ2PH7VRMR6BN7YTUCMDO5CFAU6KM6TZVZQGEG3BJFA, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.857656153 (27h26m13.897229273s) [noise=true , address=77.90.31.14:4661 , piece_id=D7ZUL2DQRR6GI5GFCOELQLZKHIVEGTJL6JLDZZWNUR4SH6TJZJLQ, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.857984952 (27h26m13.897558072s) [noise=true , address=45.91.116.5:4297 , piece_id=WECVRKLV6W5NFRRDUAH2HDJI7JTBS3U47VAIQI3BFRXCXIZVENMQ, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.857991314 (27h26m13.897564434s) [noise=true , address=45.91.116.14:4190 , piece_id=BP2ZKD44V5GDMFAZEDG2EGNMFOWLBZUFSJA7JRNSMZUQA5KM3QIA, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.858261280 (27h26m13.8978344s) [noise=true , address=77.90.31.10:4520 , piece_id=I5C7LPDZ27Z6GECAY7O2YISMVBGVZQZ6FA32T5DWGB3HUZ3G2TYQ, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.858410385 (27h26m13.897983505s) [noise=true , address=45.91.116.10:4083 , piece_id=3LO3T5XO2R4CB5YV42K4CW3A5GWL5VD35M2J54APJRSYBP5CBBLA, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.858566584 (27h26m13.898139704s) [noise=true , address=77.90.31.14:4662 , piece_id=EOV5OH4UGVSTNWT55XWBCPZ53DEZ7HFCFPVIEUS52YKTHTRRNBRQ, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.858819819 (27h26m13.898392939s) [noise=true , address=77.90.31.18:4806 , piece_id=MIA5HFPNID2IVUJZXKFSX73AUJ7G5U6DKDK4GMAYVYOSJ2VMHQDA, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.858833095 (27h26m13.898406215s) [noise=true , address=77.90.31.16:4735 , piece_id=VJU2VUREJGKHBQQKU6VXLWFFELLOE5PHKRLK5Z5BW2KHQQTSPS5A, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.859084736 (27h26m13.898657856s) [noise=true , address=77.90.31.12:4590 , piece_id=WAO64MUOLZFBMMHVOFAH4KPPN5IFYKV3EXH2DDSJZJWQ3NVAOWMA, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.859230414 (27h26m13.898803534s) [noise=true , address=45.91.116.9:4263 , piece_id=INKRGFQ6NBA6SQBDMGYCTQHPFW4E4FLBRDH7DWTNY4OHGFOU2JUA, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.859324846 (27h26m13.898897966s) [noise=true , address=77.90.31.10:4519 , piece_id=IGN4R7DRXAAS527L6RYXOJVFGYVPRYAXPQP5C72MNOD643PTD6AA, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.859532192 (27h26m13.899105312s) [noise=true , address=45.91.116.14:4189 , piece_id=Y7TBQJFIPLC36QQNC7IH6PQTJIIOCZUQQ6EU7D5TALKFJH7FI2UQ, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.859669835 (27h26m13.899242955s) [noise=true , address=45.91.116.13:4155 , piece_id=ZQIDQDRSMMQBZLMOAZKB344FGYON3EVAMYREVJSFLVJ6CUBE2KXQ, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.860040375 (27h26m13.899613495s) [noise=true , address=77.90.31.12:4592 , piece_id=EYNMBR4ZB36GZR4MFFRNHZBUP62SJNU47SGC27NVSETRHSJ2KNLQ, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:39.680046136 (27h26m15.719619246s) [upload=24152]: Next returned nil segment Sep 7 16:54:39.680063639 (27h26m15.719636749s) [upload=24152]: Waiting for error group managing segments... Sep 7 16:54:40.031506325 (27h26m16.071079445s) [noise=true , address=45.91.116.14:4189 , piece_id=Y7TBQJFIPLC36QQNC7IH6PQTJIIOCZUQQ6EU7D5TALKFJH7FI2UQ, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.042990540 (27h26m16.08256366s) [noise=true , address=77.90.31.18:4806 , piece_id=MIA5HFPNID2IVUJZXKFSX73AUJ7G5U6DKDK4GMAYVYOSJ2VMHQDA, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.074477471 (27h26m16.114050591s) [noise=true , address=77.90.31.12:4590 , piece_id=WAO64MUOLZFBMMHVOFAH4KPPN5IFYKV3EXH2DDSJZJWQ3NVAOWMA, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.127581157 (27h26m16.167154277s) [noise=true , address=45.91.116.10:4083 , piece_id=3LO3T5XO2R4CB5YV42K4CW3A5GWL5VD35M2J54APJRSYBP5CBBLA, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.134431815 (27h26m16.174004935s) [noise=true , address=77.90.31.14:4661 , piece_id=D7ZUL2DQRR6GI5GFCOELQLZKHIVEGTJL6JLDZZWNUR4SH6TJZJLQ, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.142304500 (27h26m16.18187762s) [noise=true , address=77.90.31.10:4519 , piece_id=IGN4R7DRXAAS527L6RYXOJVFGYVPRYAXPQP5C72MNOD643PTD6AA, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.157990315 (27h26m16.197563445s) [noise=true , address=45.91.116.14:4190 , piece_id=BP2ZKD44V5GDMFAZEDG2EGNMFOWLBZUFSJA7JRNSMZUQA5KM3QIA, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.171227004 (27h26m16.210800124s) [noise=true , address=77.90.31.14:4662 , piece_id=EOV5OH4UGVSTNWT55XWBCPZ53DEZ7HFCFPVIEUS52YKTHTRRNBRQ, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.182996255 (27h26m16.222569375s) [noise=true , address=45.91.116.11:4118 , piece_id=FR3OPJNV6WJ2PH7VRMR6BN7YTUCMDO5CFAU6KM6TZVZQGEG3BJFA, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.229305989 (27h26m16.268879109s) [noise=true , address=45.91.116.13:4155 , piece_id=ZQIDQDRSMMQBZLMOAZKB344FGYON3EVAMYREVJSFLVJ6CUBE2KXQ, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.249795377 (27h26m16.289368497s) [noise=true , address=77.90.31.13:4628 , piece_id=G72N6ULIGJC5USBQJBOAUT4QJ57GRK2PAN7CA32UATGQQQR5RTMQ, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.257433082 (27h26m16.297006192s) [noise=true , address=45.91.116.5:4297 , piece_id=WECVRKLV6W5NFRRDUAH2HDJI7JTBS3U47VAIQI3BFRXCXIZVENMQ, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.271532472 (27h26m16.311105582s) [noise=true , address=45.91.116.9:4263 , piece_id=INKRGFQ6NBA6SQBDMGYCTQHPFW4E4FLBRDH7DWTNY4OHGFOU2JUA, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.271557319 (27h26m16.311130439s) [seg_pos={0 0}, upload=24152]: Segment reached optimal threshold of 13 pieces. Sep 7 16:54:40.271664844 (27h26m16.311237964s) [noise=true , address=77.90.31.12:4592 , piece_id=EYNMBR4ZB36GZR4MFFRNHZBUP62SJNU47SGC27NVSETRHSJ2KNLQ, seg_pos={0 0}, upload=24152]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 7 16:54:40.271817887 (27h26m16.311391007s) [noise=true , address=77.90.31.16:4735 , piece_id=VJU2VUREJGKHBQQKU6VXLWFFELLOE5PHKRLK5Z5BW2KHQQTSPS5A, seg_pos={0 0}, upload=24152]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:36170->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:36170->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36170->77.90.31.16:4735: use of closed network connection Sep 7 16:54:40.271879244 (27h26m16.311452365s) [noise=true , address=77.90.31.10:4520 , piece_id=I5C7LPDZ27Z6GECAY7O2YISMVBGVZQZ6FA32T5DWGB3HUZ3G2TYQ, seg_pos={0 0}, upload=24152]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 7 16:54:40.271942636 (27h26m16.311515756s) [seg_pos={0 0}, upload=24152]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:54:40.272040313 (27h26m16.311613423s) [upload=24152]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:54:40.272046544 (27h26m16.311619675s) [upload=24152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:54:40.785605816 (27h26m16.825178936s) [upload=24152]: Done uploading segments... Sep 7 16:54:40.785628169 (27h26m16.825201289s) [upload=24152]: Upload finished. err: <nil> Duration: 6.609865813s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.....X.....X.................X.....................X] duration: 30.002895037s FAIL (2025-09-07 16:55:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:54:37.607783895 (27h26m13.647357015s) [upload=24152]: Starting upload Sep 7 16:54:37.607839852 (27h26m13.647412972s) [upload=24152]: Done starting upload Sep 7 16:54:37.607926227 (27h26m13.647499347s) [upload=24152]: Uploading segments... Sep 7 16:54:37.608030476 (27h26m13.647603596s) [upload=24152]: Got next segment. Inline: false Sep 7 16:54:37.608061005 (27h26m13.647634125s) [upload=24152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:54:37.608065523 (27h26m13.647638643s) [upload=24152]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:54:37.857213566 (27h26m13.896786686s) [seg_pos={0 0}, upload=24152]: Begin upload segment... Sep 7 16:54:37.857253812 (27h26m13.896826932s) [seg_pos={0 0}, upload=24152]: Done begin upload segment. Sep 7 16:54:37.857273811 (27h26m13.896846921s) [noise=true , address=77.90.31.13:4628 , piece_id=G72N6ULIGJC5USBQJBOAUT4QJ57GRK2PAN7CA32UATGQQQR5RTMQ, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.857341290 (27h26m13.89691442s) [noise=true , address=45.91.116.11:4118 , piece_id=FR3OPJNV6WJ2PH7VRMR6BN7YTUCMDO5CFAU6KM6TZVZQGEG3BJFA, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.857656153 (27h26m13.897229273s) [noise=true , address=77.90.31.14:4661 , piece_id=D7ZUL2DQRR6GI5GFCOELQLZKHIVEGTJL6JLDZZWNUR4SH6TJZJLQ, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.857984952 (27h26m13.897558072s) [noise=true , address=45.91.116.5:4297 , piece_id=WECVRKLV6W5NFRRDUAH2HDJI7JTBS3U47VAIQI3BFRXCXIZVENMQ, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.857991314 (27h26m13.897564434s) [noise=true , address=45.91.116.14:4190 , piece_id=BP2ZKD44V5GDMFAZEDG2EGNMFOWLBZUFSJA7JRNSMZUQA5KM3QIA, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.858261280 (27h26m13.8978344s) [noise=true , address=77.90.31.10:4520 , piece_id=I5C7LPDZ27Z6GECAY7O2YISMVBGVZQZ6FA32T5DWGB3HUZ3G2TYQ, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.858410385 (27h26m13.897983505s) [noise=true , address=45.91.116.10:4083 , piece_id=3LO3T5XO2R4CB5YV42K4CW3A5GWL5VD35M2J54APJRSYBP5CBBLA, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.858566584 (27h26m13.898139704s) [noise=true , address=77.90.31.14:4662 , piece_id=EOV5OH4UGVSTNWT55XWBCPZ53DEZ7HFCFPVIEUS52YKTHTRRNBRQ, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.858819819 (27h26m13.898392939s) [noise=true , address=77.90.31.18:4806 , piece_id=MIA5HFPNID2IVUJZXKFSX73AUJ7G5U6DKDK4GMAYVYOSJ2VMHQDA, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.858833095 (27h26m13.898406215s) [noise=true , address=77.90.31.16:4735 , piece_id=VJU2VUREJGKHBQQKU6VXLWFFELLOE5PHKRLK5Z5BW2KHQQTSPS5A, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.859084736 (27h26m13.898657856s) [noise=true , address=77.90.31.12:4590 , piece_id=WAO64MUOLZFBMMHVOFAH4KPPN5IFYKV3EXH2DDSJZJWQ3NVAOWMA, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.859230414 (27h26m13.898803534s) [noise=true , address=45.91.116.9:4263 , piece_id=INKRGFQ6NBA6SQBDMGYCTQHPFW4E4FLBRDH7DWTNY4OHGFOU2JUA, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.859324846 (27h26m13.898897966s) [noise=true , address=77.90.31.10:4519 , piece_id=IGN4R7DRXAAS527L6RYXOJVFGYVPRYAXPQP5C72MNOD643PTD6AA, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.859532192 (27h26m13.899105312s) [noise=true , address=45.91.116.14:4189 , piece_id=Y7TBQJFIPLC36QQNC7IH6PQTJIIOCZUQQ6EU7D5TALKFJH7FI2UQ, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.859669835 (27h26m13.899242955s) [noise=true , address=45.91.116.13:4155 , piece_id=ZQIDQDRSMMQBZLMOAZKB344FGYON3EVAMYREVJSFLVJ6CUBE2KXQ, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:37.860040375 (27h26m13.899613495s) [noise=true , address=77.90.31.12:4592 , piece_id=EYNMBR4ZB36GZR4MFFRNHZBUP62SJNU47SGC27NVSETRHSJ2KNLQ, seg_pos={0 0}, upload=24152]: Uploading piece... Sep 7 16:54:39.680046136 (27h26m15.719619246s) [upload=24152]: Next returned nil segment Sep 7 16:54:39.680063639 (27h26m15.719636749s) [upload=24152]: Waiting for error group managing segments... Sep 7 16:54:40.031506325 (27h26m16.071079445s) [noise=true , address=45.91.116.14:4189 , piece_id=Y7TBQJFIPLC36QQNC7IH6PQTJIIOCZUQQ6EU7D5TALKFJH7FI2UQ, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.042990540 (27h26m16.08256366s) [noise=true , address=77.90.31.18:4806 , piece_id=MIA5HFPNID2IVUJZXKFSX73AUJ7G5U6DKDK4GMAYVYOSJ2VMHQDA, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.074477471 (27h26m16.114050591s) [noise=true , address=77.90.31.12:4590 , piece_id=WAO64MUOLZFBMMHVOFAH4KPPN5IFYKV3EXH2DDSJZJWQ3NVAOWMA, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.127581157 (27h26m16.167154277s) [noise=true , address=45.91.116.10:4083 , piece_id=3LO3T5XO2R4CB5YV42K4CW3A5GWL5VD35M2J54APJRSYBP5CBBLA, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.134431815 (27h26m16.174004935s) [noise=true , address=77.90.31.14:4661 , piece_id=D7ZUL2DQRR6GI5GFCOELQLZKHIVEGTJL6JLDZZWNUR4SH6TJZJLQ, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.142304500 (27h26m16.18187762s) [noise=true , address=77.90.31.10:4519 , piece_id=IGN4R7DRXAAS527L6RYXOJVFGYVPRYAXPQP5C72MNOD643PTD6AA, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.157990315 (27h26m16.197563445s) [noise=true , address=45.91.116.14:4190 , piece_id=BP2ZKD44V5GDMFAZEDG2EGNMFOWLBZUFSJA7JRNSMZUQA5KM3QIA, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.171227004 (27h26m16.210800124s) [noise=true , address=77.90.31.14:4662 , piece_id=EOV5OH4UGVSTNWT55XWBCPZ53DEZ7HFCFPVIEUS52YKTHTRRNBRQ, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.182996255 (27h26m16.222569375s) [noise=true , address=45.91.116.11:4118 , piece_id=FR3OPJNV6WJ2PH7VRMR6BN7YTUCMDO5CFAU6KM6TZVZQGEG3BJFA, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.229305989 (27h26m16.268879109s) [noise=true , address=45.91.116.13:4155 , piece_id=ZQIDQDRSMMQBZLMOAZKB344FGYON3EVAMYREVJSFLVJ6CUBE2KXQ, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.249795377 (27h26m16.289368497s) [noise=true , address=77.90.31.13:4628 , piece_id=G72N6ULIGJC5USBQJBOAUT4QJ57GRK2PAN7CA32UATGQQQR5RTMQ, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.257433082 (27h26m16.297006192s) [noise=true , address=45.91.116.5:4297 , piece_id=WECVRKLV6W5NFRRDUAH2HDJI7JTBS3U47VAIQI3BFRXCXIZVENMQ, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.271532472 (27h26m16.311105582s) [noise=true , address=45.91.116.9:4263 , piece_id=INKRGFQ6NBA6SQBDMGYCTQHPFW4E4FLBRDH7DWTNY4OHGFOU2JUA, seg_pos={0 0}, upload=24152]: Done uploading piece. err: <nil> Sep 7 16:54:40.271557319 (27h26m16.311130439s) [seg_pos={0 0}, upload=24152]: Segment reached optimal threshold of 13 pieces. Sep 7 16:54:40.271664844 (27h26m16.311237964s) [noise=true , address=77.90.31.12:4592 , piece_id=EYNMBR4ZB36GZR4MFFRNHZBUP62SJNU47SGC27NVSETRHSJ2KNLQ, seg_pos={0 0}, upload=24152]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Sep 7 16:54:40.271817887 (27h26m16.311391007s) [noise=true , address=77.90.31.16:4735 , piece_id=VJU2VUREJGKHBQQKU6VXLWFFELLOE5PHKRLK5Z5BW2KHQQTSPS5A, seg_pos={0 0}, upload=24152]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:36170->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:36170->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36170->77.90.31.16:4735: use of closed network connection Sep 7 16:54:40.271879244 (27h26m16.311452365s) [noise=true , address=77.90.31.10:4520 , piece_id=I5C7LPDZ27Z6GECAY7O2YISMVBGVZQZ6FA32T5DWGB3HUZ3G2TYQ, seg_pos={0 0}, upload=24152]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Sep 7 16:54:40.271942636 (27h26m16.311515756s) [seg_pos={0 0}, upload=24152]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:54:40.272040313 (27h26m16.311613423s) [upload=24152]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:54:40.272046544 (27h26m16.311619675s) [upload=24152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:54:40.785605816 (27h26m16.825178936s) [upload=24152]: Done uploading segments... Sep 7 16:54:40.785628169 (27h26m16.825201289s) [upload=24152]: Upload finished. err: <nil> Duration: 30.002914645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.....X.....X.................X.....................] duration: 6.480446686s FAIL (2025-09-07 16:54:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 6.480528072s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.....X.....X.................X....................] duration: 10.126681917s FAIL (2025-09-07 16:53:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 10.126707968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.....X.....X.................X...................] duration: 6.912063107s FAIL (2025-09-07 16:52:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 6.912092071s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.....X.....X.................X..................] duration: 7.25942397s FAIL (2025-09-07 16:50:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 7.259454298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X...........X.....X.....X.................X.................] duration: 6.905359719s FAIL (2025-09-07 16:50:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 6.905391901s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X...........X.....X.....X.................X................] duration: 6.534306167s FAIL (2025-09-07 16:49:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 6.53433413s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X...........X.....X.....X.................X...............] duration: 8.960337416s FAIL (2025-09-07 16:49:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 8.960357544s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..X...........X.....X.....X.................X..............] duration: 24.08761816s FAIL (2025-09-07 16:48:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 24.087652484s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..X...........X.....X.....X.................X.............] duration: 11.865139035s FAIL (2025-09-07 16:47:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 11.865169913s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.X..X...........X.....X.....X.................X............] duration: 6.696320207s FAIL (2025-09-07 16:46:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 6.696345276s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.X..X...........X.....X.....X.................X...........] duration: 12.188157532s FAIL (2025-09-07 16:46:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 12.188183432s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.X.X..X...........X.....X.....X.................X..........] duration: 6.474643779s FAIL (2025-09-07 16:45:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 6.474678275s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.X.X..X...........X.....X.....X.................X.........] duration: 6.404426286s FAIL (2025-09-07 16:45:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 6.404452927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.X.X..X...........X.....X.....X.................X........] duration: 6.312641466s FAIL (2025-09-07 16:44:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 6.312672586s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X.X.X..X...........X.....X.....X.................X.......] duration: 6.664349145s FAIL (2025-09-07 16:43:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 6.664376569s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X.X.X..X...........X.....X.....X.................X......] duration: 6.437365457s FAIL (2025-09-07 16:42:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 6.43738816s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X.X.X..X...........X.....X.....X.................X.....] duration: 6.099510208s FAIL (2025-09-07 16:41:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 6.099539906s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X.X.X..X...........X.....X.....X.................X....] duration: 6.563212935s FAIL (2025-09-07 16:40:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 6.563242242s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X......X.X.X..X...........X.....X.....X.................X...] duration: 9.266045012s FAIL (2025-09-07 16:39:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 9.266072605s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X......X.X.X..X...........X.....X.....X.................X..] duration: 9.877526859s FAIL (2025-09-07 16:39:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 9.877552548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.X......X.X.X..X...........X.....X.....X.................X.] duration: 8.074185976s FAIL (2025-09-07 16:38:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 8.074202598s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.X......X.X.X..X...........X.....X.....X.................X] duration: 30.002264184s FAIL (2025-09-07 16:38:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:38:16.630194789 (27h9m52.669767909s) [upload=23908]: Starting upload Sep 7 16:38:16.630247700 (27h9m52.66982082s) [upload=23908]: Done starting upload Sep 7 16:38:16.630333163 (27h9m52.669906283s) [upload=23908]: Uploading segments... Sep 7 16:38:16.630419068 (27h9m52.669992188s) [upload=23908]: Got next segment. Inline: false Sep 7 16:38:16.630455558 (27h9m52.670028678s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:38:16.630459766 (27h9m52.670032886s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:38:16.894297276 (27h9m52.933870396s) [seg_pos={0 0}, upload=23908]: Begin upload segment... Sep 7 16:38:16.894339948 (27h9m52.933913058s) [seg_pos={0 0}, upload=23908]: Done begin upload segment. Sep 7 16:38:16.894363363 (27h9m52.933936483s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894398219 (27h9m52.933971329s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894779619 (27h9m52.934352739s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.894828423 (27h9m52.934401543s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895050417 (27h9m52.934623537s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895220212 (27h9m52.934793322s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895318631 (27h9m52.934891751s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895514415 (27h9m52.935087535s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895591062 (27h9m52.935164182s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.895834508 (27h9m52.935407628s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896043597 (27h9m52.935616718s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896260483 (27h9m52.935833603s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896492928 (27h9m52.936066048s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896716425 (27h9m52.936289545s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.896903954 (27h9m52.936477074s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:16.897098356 (27h9m52.936671476s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Uploading piece... Sep 7 16:38:18.736377025 (27h9m54.775950135s) [upload=23908]: Next returned nil segment Sep 7 16:38:18.736392915 (27h9m54.775966035s) [upload=23908]: Waiting for error group managing segments... Sep 7 16:38:19.032044874 (27h9m55.071617984s) [noise=true , address=45.91.116.11:4119 , piece_id=5XNOP6U5S6YQMXELQNYHFUF625BHKEGQ7E6JSZLA3XXD72AAMHUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.032460549 (27h9m55.072033669s) [noise=true , address=45.91.116.6:4973 , piece_id=P7NZVOLQSSQVUQFST7WQXURUCQTGHDQB32LIMDSOMEJ2PFFWPMNA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.061943616 (27h9m55.101516736s) [noise=true , address=45.91.116.8:4048 , piece_id=FQB4LSSYCPHBMOKXQPP5QZYVKQ7S7YNIPBBAFJP6GYLHA2LLDFIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.062151333 (27h9m55.101724453s) [noise=true , address=45.91.116.11:4118 , piece_id=D234XBDL6B57FLBHADJOTHZVT7D3NLNXIRDWWQLIDAS4X2GJHSRA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.082344553 (27h9m55.121917673s) [noise=true , address=45.91.116.10:4083 , piece_id=2B5SUW3P5LXL2B4UZDHALAE5OBFXEW2DJMDFFPWJB3RM73UOPNXA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.128172376 (27h9m55.167745516s) [noise=true , address=45.91.116.7:4011 , piece_id=HLTLERPP7RONZ2N6JNDZBIKAJXK5TQSYZWVEN7VM7AJNOXKO6MXQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.138501050 (27h9m55.17807421s) [noise=true , address=45.91.116.10:4082 , piece_id=Q7UKXT6KACLYEMKZB2FNU7HPUUY23QUILCGBNGZRNGRVOSC5LETQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.165670891 (27h9m55.205244011s) [noise=true , address=45.91.116.14:4189 , piece_id=VVC2KI56BBGGNBNRU6PUFAZHTKRXBRDBU3BLAKZWHH7NY45OSJDA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.167553083 (27h9m55.207126203s) [noise=true , address=45.91.116.12:4225 , piece_id=7ZYRGGJTWKKR5FH4C2CJNYQFAXUIFHC7DP6KT76MEGRGUX52REUQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.209017886 (27h9m55.248590996s) [noise=true , address=45.91.116.9:4262 , piece_id=7DHJXHDZ3DR2GVU4ZIFK457JGG7WJZTX7VW4RGI4QXUF7DTYQKGA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.227817470 (27h9m55.26739059s) [noise=true , address=45.91.116.13:4155 , piece_id=KWUKFA7YPD7UYEAOMDUBZ7OQVRVD7KZAUCXHDIRLIJFVCDGWACCA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.231214820 (27h9m55.270787941s) [noise=true , address=77.90.31.13:4627 , piece_id=JUZ62GG2QI46Q2K4CPDKHOEMB7WLXLB26MGIFFY2Z5OHNCACI4OA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282222485 (27h9m55.321795595s) [noise=true , address=45.91.116.13:4153 , piece_id=I4CXOHXVPJXAZPQWOQJ46624JYJSZ7JWN5VSTOJMU5T7GORIS75A, seg_pos={0 0}, upload=23908]: Done uploading piece. err: <nil> Sep 7 16:38:19.282238807 (27h9m55.321811927s) [seg_pos={0 0}, upload=23908]: Segment reached optimal threshold of 13 pieces. Sep 7 16:38:19.282375028 (27h9m55.321948148s) [noise=true , address=77.90.31.13:4626 , piece_id=VKFCJBAZ2EAY3VO63PN4SPSSI6QE7LGUML4IM6ATFOGMAVXIXVRQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:48294->77.90.31.13:4626: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:38:19.282470981 (27h9m55.322044101s) [noise=true , address=77.90.31.17:4772 , piece_id=VHMLNOVVRPBARLK3PFURXBTJBR2PQJI6HZ6F4RRKQC72RCNZYMPA, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39638->77.90.31.17:4772: use of closed network connection Sep 7 16:38:19.282540234 (27h9m55.322113354s) [noise=true , address=45.91.116.6:4975 , piece_id=OL5Y5ABWICMMZZJGHUMQE2YUTEYJCFBFV6EC4ZNOHJHTC7AWAJIQ, seg_pos={0 0}, upload=23908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49508->45.91.116.6:4975: use of closed network connection Sep 7 16:38:19.282558758 (27h9m55.322131879s) [seg_pos={0 0}, upload=23908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:38:19.282622912 (27h9m55.322196032s) [upload=23908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:38:19.282626709 (27h9m55.322199829s) [upload=23908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:38:19.776542220 (27h9m55.81611534s) [upload=23908]: Done uploading segments... Sep 7 16:38:19.776561106 (27h9m55.816134226s) [upload=23908]: Upload finished. err: <nil> Duration: 30.002292467s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X.X......X.X.X..X...........X.....X.....X.................] duration: 6.414215517s FAIL (2025-09-07 16:37:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:24:47.729359874 (26h56m23.768932994s) [upload=23710]: Starting upload Sep 7 16:24:47.729425129 (26h56m23.768998279s) [upload=23710]: Done starting upload Sep 7 16:24:47.729547583 (26h56m23.769120703s) [upload=23710]: Uploading segments... Sep 7 16:24:47.729893155 (26h56m23.769466275s) [upload=23710]: Got next segment. Inline: false Sep 7 16:24:47.729932419 (26h56m23.76950554s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:24:47.729936908 (26h56m23.769510028s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:24:48.017044215 (26h56m24.056617336s) [seg_pos={0 0}, upload=23710]: Begin upload segment... Sep 7 16:24:48.017099892 (26h56m24.056673012s) [seg_pos={0 0}, upload=23710]: Done begin upload segment. Sep 7 16:24:48.017123026 (26h56m24.056696146s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017169436 (26h56m24.056742556s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017486843 (26h56m24.057059963s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017740939 (26h56m24.057314059s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017838135 (26h56m24.057411255s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018088304 (26h56m24.057661414s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018122940 (26h56m24.05769606s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018402465 (26h56m24.057975585s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018482167 (26h56m24.058055287s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018681529 (26h56m24.058254649s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018912802 (26h56m24.058485912s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019116080 (26h56m24.0586892s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019332244 (26h56m24.058905354s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019547526 (26h56m24.059120646s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019749592 (26h56m24.059322712s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019935007 (26h56m24.059508117s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:49.742054032 (26h56m25.781627152s) [upload=23710]: Next returned nil segment Sep 7 16:24:49.742070412 (26h56m25.781643522s) [upload=23710]: Waiting for error group managing segments... Sep 7 16:24:50.156729825 (26h56m26.196302945s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.179916844 (26h56m26.219489964s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.190874311 (26h56m26.230447431s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.206966775 (26h56m26.246539895s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.233097747 (26h56m26.272670867s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.259133567 (26h56m26.298706687s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.283376256 (26h56m26.322949376s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.294991472 (26h56m26.334564592s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.313999864 (26h56m26.353572985s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.327448961 (26h56m26.367022081s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.330061599 (26h56m26.369634719s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.335870615 (26h56m26.375443735s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350484780 (26h56m26.3900579s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350508866 (26h56m26.390081976s) [seg_pos={0 0}, upload=23710]: Segment reached optimal threshold of 13 pieces. Sep 7 16:24:50.350613436 (26h56m26.390186556s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350672970 (26h56m26.39024609s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350710802 (26h56m26.390283922s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:44398->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:24:50.350745478 (26h56m26.390318599s) [seg_pos={0 0}, upload=23710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:24:50.350811014 (26h56m26.390384135s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:24:50.350814852 (26h56m26.390387972s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:24:50.790876341 (26h56m26.830449461s) [upload=23710]: Done uploading segments... Sep 7 16:24:50.790895568 (26h56m26.830468688s) [upload=23710]: Upload finished. err: <nil> Duration: 6.414244271s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X..X.X......X.X.X..X...........X.....X.....X................] duration: 11.630099012s FAIL (2025-09-07 16:37:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:24:47.729359874 (26h56m23.768932994s) [upload=23710]: Starting upload Sep 7 16:24:47.729425129 (26h56m23.768998279s) [upload=23710]: Done starting upload Sep 7 16:24:47.729547583 (26h56m23.769120703s) [upload=23710]: Uploading segments... Sep 7 16:24:47.729893155 (26h56m23.769466275s) [upload=23710]: Got next segment. Inline: false Sep 7 16:24:47.729932419 (26h56m23.76950554s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:24:47.729936908 (26h56m23.769510028s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:24:48.017044215 (26h56m24.056617336s) [seg_pos={0 0}, upload=23710]: Begin upload segment... Sep 7 16:24:48.017099892 (26h56m24.056673012s) [seg_pos={0 0}, upload=23710]: Done begin upload segment. Sep 7 16:24:48.017123026 (26h56m24.056696146s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017169436 (26h56m24.056742556s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017486843 (26h56m24.057059963s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017740939 (26h56m24.057314059s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017838135 (26h56m24.057411255s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018088304 (26h56m24.057661414s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018122940 (26h56m24.05769606s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018402465 (26h56m24.057975585s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018482167 (26h56m24.058055287s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018681529 (26h56m24.058254649s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018912802 (26h56m24.058485912s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019116080 (26h56m24.0586892s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019332244 (26h56m24.058905354s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019547526 (26h56m24.059120646s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019749592 (26h56m24.059322712s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019935007 (26h56m24.059508117s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:49.742054032 (26h56m25.781627152s) [upload=23710]: Next returned nil segment Sep 7 16:24:49.742070412 (26h56m25.781643522s) [upload=23710]: Waiting for error group managing segments... Sep 7 16:24:50.156729825 (26h56m26.196302945s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.179916844 (26h56m26.219489964s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.190874311 (26h56m26.230447431s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.206966775 (26h56m26.246539895s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.233097747 (26h56m26.272670867s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.259133567 (26h56m26.298706687s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.283376256 (26h56m26.322949376s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.294991472 (26h56m26.334564592s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.313999864 (26h56m26.353572985s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.327448961 (26h56m26.367022081s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.330061599 (26h56m26.369634719s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.335870615 (26h56m26.375443735s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350484780 (26h56m26.3900579s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350508866 (26h56m26.390081976s) [seg_pos={0 0}, upload=23710]: Segment reached optimal threshold of 13 pieces. Sep 7 16:24:50.350613436 (26h56m26.390186556s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350672970 (26h56m26.39024609s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350710802 (26h56m26.390283922s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:44398->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:24:50.350745478 (26h56m26.390318599s) [seg_pos={0 0}, upload=23710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:24:50.350811014 (26h56m26.390384135s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:24:50.350814852 (26h56m26.390387972s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:24:50.790876341 (26h56m26.830449461s) [upload=23710]: Done uploading segments... Sep 7 16:24:50.790895568 (26h56m26.830468688s) [upload=23710]: Upload finished. err: <nil> Duration: 11.630129151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X..X.X......X.X.X..X...........X.....X.....X...............] duration: 6.716761704s FAIL (2025-09-07 16:36:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:24:47.729359874 (26h56m23.768932994s) [upload=23710]: Starting upload Sep 7 16:24:47.729425129 (26h56m23.768998279s) [upload=23710]: Done starting upload Sep 7 16:24:47.729547583 (26h56m23.769120703s) [upload=23710]: Uploading segments... Sep 7 16:24:47.729893155 (26h56m23.769466275s) [upload=23710]: Got next segment. Inline: false Sep 7 16:24:47.729932419 (26h56m23.76950554s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:24:47.729936908 (26h56m23.769510028s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:24:48.017044215 (26h56m24.056617336s) [seg_pos={0 0}, upload=23710]: Begin upload segment... Sep 7 16:24:48.017099892 (26h56m24.056673012s) [seg_pos={0 0}, upload=23710]: Done begin upload segment. Sep 7 16:24:48.017123026 (26h56m24.056696146s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017169436 (26h56m24.056742556s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017486843 (26h56m24.057059963s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017740939 (26h56m24.057314059s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017838135 (26h56m24.057411255s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018088304 (26h56m24.057661414s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018122940 (26h56m24.05769606s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018402465 (26h56m24.057975585s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018482167 (26h56m24.058055287s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018681529 (26h56m24.058254649s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018912802 (26h56m24.058485912s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019116080 (26h56m24.0586892s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019332244 (26h56m24.058905354s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019547526 (26h56m24.059120646s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019749592 (26h56m24.059322712s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019935007 (26h56m24.059508117s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:49.742054032 (26h56m25.781627152s) [upload=23710]: Next returned nil segment Sep 7 16:24:49.742070412 (26h56m25.781643522s) [upload=23710]: Waiting for error group managing segments... Sep 7 16:24:50.156729825 (26h56m26.196302945s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.179916844 (26h56m26.219489964s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.190874311 (26h56m26.230447431s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.206966775 (26h56m26.246539895s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.233097747 (26h56m26.272670867s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.259133567 (26h56m26.298706687s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.283376256 (26h56m26.322949376s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.294991472 (26h56m26.334564592s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.313999864 (26h56m26.353572985s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.327448961 (26h56m26.367022081s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.330061599 (26h56m26.369634719s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.335870615 (26h56m26.375443735s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350484780 (26h56m26.3900579s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350508866 (26h56m26.390081976s) [seg_pos={0 0}, upload=23710]: Segment reached optimal threshold of 13 pieces. Sep 7 16:24:50.350613436 (26h56m26.390186556s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350672970 (26h56m26.39024609s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350710802 (26h56m26.390283922s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:44398->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:24:50.350745478 (26h56m26.390318599s) [seg_pos={0 0}, upload=23710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:24:50.350811014 (26h56m26.390384135s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:24:50.350814852 (26h56m26.390387972s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:24:50.790876341 (26h56m26.830449461s) [upload=23710]: Done uploading segments... Sep 7 16:24:50.790895568 (26h56m26.830468688s) [upload=23710]: Upload finished. err: <nil> Duration: 6.716789739s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X..X.X......X.X.X..X...........X.....X.....X..............] duration: 6.505214296s FAIL (2025-09-07 16:35:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:24:47.729359874 (26h56m23.768932994s) [upload=23710]: Starting upload Sep 7 16:24:47.729425129 (26h56m23.768998279s) [upload=23710]: Done starting upload Sep 7 16:24:47.729547583 (26h56m23.769120703s) [upload=23710]: Uploading segments... Sep 7 16:24:47.729893155 (26h56m23.769466275s) [upload=23710]: Got next segment. Inline: false Sep 7 16:24:47.729932419 (26h56m23.76950554s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:24:47.729936908 (26h56m23.769510028s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:24:48.017044215 (26h56m24.056617336s) [seg_pos={0 0}, upload=23710]: Begin upload segment... Sep 7 16:24:48.017099892 (26h56m24.056673012s) [seg_pos={0 0}, upload=23710]: Done begin upload segment. Sep 7 16:24:48.017123026 (26h56m24.056696146s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017169436 (26h56m24.056742556s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017486843 (26h56m24.057059963s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017740939 (26h56m24.057314059s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017838135 (26h56m24.057411255s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018088304 (26h56m24.057661414s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018122940 (26h56m24.05769606s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018402465 (26h56m24.057975585s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018482167 (26h56m24.058055287s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018681529 (26h56m24.058254649s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018912802 (26h56m24.058485912s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019116080 (26h56m24.0586892s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019332244 (26h56m24.058905354s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019547526 (26h56m24.059120646s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019749592 (26h56m24.059322712s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019935007 (26h56m24.059508117s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:49.742054032 (26h56m25.781627152s) [upload=23710]: Next returned nil segment Sep 7 16:24:49.742070412 (26h56m25.781643522s) [upload=23710]: Waiting for error group managing segments... Sep 7 16:24:50.156729825 (26h56m26.196302945s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.179916844 (26h56m26.219489964s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.190874311 (26h56m26.230447431s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.206966775 (26h56m26.246539895s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.233097747 (26h56m26.272670867s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.259133567 (26h56m26.298706687s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.283376256 (26h56m26.322949376s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.294991472 (26h56m26.334564592s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.313999864 (26h56m26.353572985s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.327448961 (26h56m26.367022081s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.330061599 (26h56m26.369634719s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.335870615 (26h56m26.375443735s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350484780 (26h56m26.3900579s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350508866 (26h56m26.390081976s) [seg_pos={0 0}, upload=23710]: Segment reached optimal threshold of 13 pieces. Sep 7 16:24:50.350613436 (26h56m26.390186556s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350672970 (26h56m26.39024609s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350710802 (26h56m26.390283922s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:44398->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:24:50.350745478 (26h56m26.390318599s) [seg_pos={0 0}, upload=23710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:24:50.350811014 (26h56m26.390384135s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:24:50.350814852 (26h56m26.390387972s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:24:50.790876341 (26h56m26.830449461s) [upload=23710]: Done uploading segments... Sep 7 16:24:50.790895568 (26h56m26.830468688s) [upload=23710]: Upload finished. err: <nil> Duration: 6.50524284s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X..X.X......X.X.X..X...........X.....X.....X.............] duration: 8.214435469s FAIL (2025-09-07 16:34:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:24:47.729359874 (26h56m23.768932994s) [upload=23710]: Starting upload Sep 7 16:24:47.729425129 (26h56m23.768998279s) [upload=23710]: Done starting upload Sep 7 16:24:47.729547583 (26h56m23.769120703s) [upload=23710]: Uploading segments... Sep 7 16:24:47.729893155 (26h56m23.769466275s) [upload=23710]: Got next segment. Inline: false Sep 7 16:24:47.729932419 (26h56m23.76950554s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:24:47.729936908 (26h56m23.769510028s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:24:48.017044215 (26h56m24.056617336s) [seg_pos={0 0}, upload=23710]: Begin upload segment... Sep 7 16:24:48.017099892 (26h56m24.056673012s) [seg_pos={0 0}, upload=23710]: Done begin upload segment. Sep 7 16:24:48.017123026 (26h56m24.056696146s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017169436 (26h56m24.056742556s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017486843 (26h56m24.057059963s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017740939 (26h56m24.057314059s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017838135 (26h56m24.057411255s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018088304 (26h56m24.057661414s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018122940 (26h56m24.05769606s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018402465 (26h56m24.057975585s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018482167 (26h56m24.058055287s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018681529 (26h56m24.058254649s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018912802 (26h56m24.058485912s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019116080 (26h56m24.0586892s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019332244 (26h56m24.058905354s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019547526 (26h56m24.059120646s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019749592 (26h56m24.059322712s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019935007 (26h56m24.059508117s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:49.742054032 (26h56m25.781627152s) [upload=23710]: Next returned nil segment Sep 7 16:24:49.742070412 (26h56m25.781643522s) [upload=23710]: Waiting for error group managing segments... Sep 7 16:24:50.156729825 (26h56m26.196302945s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.179916844 (26h56m26.219489964s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.190874311 (26h56m26.230447431s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.206966775 (26h56m26.246539895s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.233097747 (26h56m26.272670867s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.259133567 (26h56m26.298706687s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.283376256 (26h56m26.322949376s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.294991472 (26h56m26.334564592s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.313999864 (26h56m26.353572985s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.327448961 (26h56m26.367022081s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.330061599 (26h56m26.369634719s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.335870615 (26h56m26.375443735s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350484780 (26h56m26.3900579s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350508866 (26h56m26.390081976s) [seg_pos={0 0}, upload=23710]: Segment reached optimal threshold of 13 pieces. Sep 7 16:24:50.350613436 (26h56m26.390186556s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350672970 (26h56m26.39024609s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350710802 (26h56m26.390283922s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:44398->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:24:50.350745478 (26h56m26.390318599s) [seg_pos={0 0}, upload=23710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:24:50.350811014 (26h56m26.390384135s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:24:50.350814852 (26h56m26.390387972s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:24:50.790876341 (26h56m26.830449461s) [upload=23710]: Done uploading segments... Sep 7 16:24:50.790895568 (26h56m26.830468688s) [upload=23710]: Upload finished. err: <nil> Duration: 8.2144619s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X..X.X......X.X.X..X...........X.....X.....X............] duration: 9.328759399s FAIL (2025-09-07 16:34:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:24:47.729359874 (26h56m23.768932994s) [upload=23710]: Starting upload Sep 7 16:24:47.729425129 (26h56m23.768998279s) [upload=23710]: Done starting upload Sep 7 16:24:47.729547583 (26h56m23.769120703s) [upload=23710]: Uploading segments... Sep 7 16:24:47.729893155 (26h56m23.769466275s) [upload=23710]: Got next segment. Inline: false Sep 7 16:24:47.729932419 (26h56m23.76950554s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:24:47.729936908 (26h56m23.769510028s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:24:48.017044215 (26h56m24.056617336s) [seg_pos={0 0}, upload=23710]: Begin upload segment... Sep 7 16:24:48.017099892 (26h56m24.056673012s) [seg_pos={0 0}, upload=23710]: Done begin upload segment. Sep 7 16:24:48.017123026 (26h56m24.056696146s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017169436 (26h56m24.056742556s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017486843 (26h56m24.057059963s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017740939 (26h56m24.057314059s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017838135 (26h56m24.057411255s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018088304 (26h56m24.057661414s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018122940 (26h56m24.05769606s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018402465 (26h56m24.057975585s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018482167 (26h56m24.058055287s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018681529 (26h56m24.058254649s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018912802 (26h56m24.058485912s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019116080 (26h56m24.0586892s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019332244 (26h56m24.058905354s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019547526 (26h56m24.059120646s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019749592 (26h56m24.059322712s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019935007 (26h56m24.059508117s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:49.742054032 (26h56m25.781627152s) [upload=23710]: Next returned nil segment Sep 7 16:24:49.742070412 (26h56m25.781643522s) [upload=23710]: Waiting for error group managing segments... Sep 7 16:24:50.156729825 (26h56m26.196302945s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.179916844 (26h56m26.219489964s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.190874311 (26h56m26.230447431s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.206966775 (26h56m26.246539895s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.233097747 (26h56m26.272670867s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.259133567 (26h56m26.298706687s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.283376256 (26h56m26.322949376s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.294991472 (26h56m26.334564592s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.313999864 (26h56m26.353572985s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.327448961 (26h56m26.367022081s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.330061599 (26h56m26.369634719s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.335870615 (26h56m26.375443735s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350484780 (26h56m26.3900579s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350508866 (26h56m26.390081976s) [seg_pos={0 0}, upload=23710]: Segment reached optimal threshold of 13 pieces. Sep 7 16:24:50.350613436 (26h56m26.390186556s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350672970 (26h56m26.39024609s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350710802 (26h56m26.390283922s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:44398->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:24:50.350745478 (26h56m26.390318599s) [seg_pos={0 0}, upload=23710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:24:50.350811014 (26h56m26.390384135s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:24:50.350814852 (26h56m26.390387972s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:24:50.790876341 (26h56m26.830449461s) [upload=23710]: Done uploading segments... Sep 7 16:24:50.790895568 (26h56m26.830468688s) [upload=23710]: Upload finished. err: <nil> Duration: 9.328810607s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X..X.X......X.X.X..X...........X.....X.....X...........] duration: 16.472435959s FAIL (2025-09-07 16:33:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:24:47.729359874 (26h56m23.768932994s) [upload=23710]: Starting upload Sep 7 16:24:47.729425129 (26h56m23.768998279s) [upload=23710]: Done starting upload Sep 7 16:24:47.729547583 (26h56m23.769120703s) [upload=23710]: Uploading segments... Sep 7 16:24:47.729893155 (26h56m23.769466275s) [upload=23710]: Got next segment. Inline: false Sep 7 16:24:47.729932419 (26h56m23.76950554s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:24:47.729936908 (26h56m23.769510028s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:24:48.017044215 (26h56m24.056617336s) [seg_pos={0 0}, upload=23710]: Begin upload segment... Sep 7 16:24:48.017099892 (26h56m24.056673012s) [seg_pos={0 0}, upload=23710]: Done begin upload segment. Sep 7 16:24:48.017123026 (26h56m24.056696146s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017169436 (26h56m24.056742556s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017486843 (26h56m24.057059963s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017740939 (26h56m24.057314059s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017838135 (26h56m24.057411255s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018088304 (26h56m24.057661414s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018122940 (26h56m24.05769606s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018402465 (26h56m24.057975585s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018482167 (26h56m24.058055287s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018681529 (26h56m24.058254649s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018912802 (26h56m24.058485912s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019116080 (26h56m24.0586892s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019332244 (26h56m24.058905354s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019547526 (26h56m24.059120646s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019749592 (26h56m24.059322712s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019935007 (26h56m24.059508117s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:49.742054032 (26h56m25.781627152s) [upload=23710]: Next returned nil segment Sep 7 16:24:49.742070412 (26h56m25.781643522s) [upload=23710]: Waiting for error group managing segments... Sep 7 16:24:50.156729825 (26h56m26.196302945s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.179916844 (26h56m26.219489964s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.190874311 (26h56m26.230447431s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.206966775 (26h56m26.246539895s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.233097747 (26h56m26.272670867s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.259133567 (26h56m26.298706687s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.283376256 (26h56m26.322949376s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.294991472 (26h56m26.334564592s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.313999864 (26h56m26.353572985s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.327448961 (26h56m26.367022081s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.330061599 (26h56m26.369634719s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.335870615 (26h56m26.375443735s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350484780 (26h56m26.3900579s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350508866 (26h56m26.390081976s) [seg_pos={0 0}, upload=23710]: Segment reached optimal threshold of 13 pieces. Sep 7 16:24:50.350613436 (26h56m26.390186556s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350672970 (26h56m26.39024609s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350710802 (26h56m26.390283922s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:44398->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:24:50.350745478 (26h56m26.390318599s) [seg_pos={0 0}, upload=23710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:24:50.350811014 (26h56m26.390384135s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:24:50.350814852 (26h56m26.390387972s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:24:50.790876341 (26h56m26.830449461s) [upload=23710]: Done uploading segments... Sep 7 16:24:50.790895568 (26h56m26.830468688s) [upload=23710]: Upload finished. err: <nil> Duration: 16.472484051s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......X..X.X......X.X.X..X...........X.....X.....X..........] duration: 10.728865027s FAIL (2025-09-07 16:33:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:24:47.729359874 (26h56m23.768932994s) [upload=23710]: Starting upload Sep 7 16:24:47.729425129 (26h56m23.768998279s) [upload=23710]: Done starting upload Sep 7 16:24:47.729547583 (26h56m23.769120703s) [upload=23710]: Uploading segments... Sep 7 16:24:47.729893155 (26h56m23.769466275s) [upload=23710]: Got next segment. Inline: false Sep 7 16:24:47.729932419 (26h56m23.76950554s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:24:47.729936908 (26h56m23.769510028s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:24:48.017044215 (26h56m24.056617336s) [seg_pos={0 0}, upload=23710]: Begin upload segment... Sep 7 16:24:48.017099892 (26h56m24.056673012s) [seg_pos={0 0}, upload=23710]: Done begin upload segment. Sep 7 16:24:48.017123026 (26h56m24.056696146s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017169436 (26h56m24.056742556s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017486843 (26h56m24.057059963s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017740939 (26h56m24.057314059s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017838135 (26h56m24.057411255s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018088304 (26h56m24.057661414s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018122940 (26h56m24.05769606s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018402465 (26h56m24.057975585s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018482167 (26h56m24.058055287s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018681529 (26h56m24.058254649s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018912802 (26h56m24.058485912s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019116080 (26h56m24.0586892s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019332244 (26h56m24.058905354s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019547526 (26h56m24.059120646s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019749592 (26h56m24.059322712s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019935007 (26h56m24.059508117s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:49.742054032 (26h56m25.781627152s) [upload=23710]: Next returned nil segment Sep 7 16:24:49.742070412 (26h56m25.781643522s) [upload=23710]: Waiting for error group managing segments... Sep 7 16:24:50.156729825 (26h56m26.196302945s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.179916844 (26h56m26.219489964s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.190874311 (26h56m26.230447431s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.206966775 (26h56m26.246539895s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.233097747 (26h56m26.272670867s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.259133567 (26h56m26.298706687s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.283376256 (26h56m26.322949376s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.294991472 (26h56m26.334564592s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.313999864 (26h56m26.353572985s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.327448961 (26h56m26.367022081s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.330061599 (26h56m26.369634719s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.335870615 (26h56m26.375443735s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350484780 (26h56m26.3900579s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350508866 (26h56m26.390081976s) [seg_pos={0 0}, upload=23710]: Segment reached optimal threshold of 13 pieces. Sep 7 16:24:50.350613436 (26h56m26.390186556s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350672970 (26h56m26.39024609s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350710802 (26h56m26.390283922s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:44398->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:24:50.350745478 (26h56m26.390318599s) [seg_pos={0 0}, upload=23710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:24:50.350811014 (26h56m26.390384135s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:24:50.350814852 (26h56m26.390387972s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:24:50.790876341 (26h56m26.830449461s) [upload=23710]: Done uploading segments... Sep 7 16:24:50.790895568 (26h56m26.830468688s) [upload=23710]: Upload finished. err: <nil> Duration: 10.728898502s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X......X..X.X......X.X.X..X...........X.....X.....X.........] duration: 7.270572861s FAIL (2025-09-07 16:32:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:24:47.729359874 (26h56m23.768932994s) [upload=23710]: Starting upload Sep 7 16:24:47.729425129 (26h56m23.768998279s) [upload=23710]: Done starting upload Sep 7 16:24:47.729547583 (26h56m23.769120703s) [upload=23710]: Uploading segments... Sep 7 16:24:47.729893155 (26h56m23.769466275s) [upload=23710]: Got next segment. Inline: false Sep 7 16:24:47.729932419 (26h56m23.76950554s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:24:47.729936908 (26h56m23.769510028s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:24:48.017044215 (26h56m24.056617336s) [seg_pos={0 0}, upload=23710]: Begin upload segment... Sep 7 16:24:48.017099892 (26h56m24.056673012s) [seg_pos={0 0}, upload=23710]: Done begin upload segment. Sep 7 16:24:48.017123026 (26h56m24.056696146s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017169436 (26h56m24.056742556s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017486843 (26h56m24.057059963s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017740939 (26h56m24.057314059s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017838135 (26h56m24.057411255s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018088304 (26h56m24.057661414s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018122940 (26h56m24.05769606s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018402465 (26h56m24.057975585s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018482167 (26h56m24.058055287s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018681529 (26h56m24.058254649s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018912802 (26h56m24.058485912s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019116080 (26h56m24.0586892s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019332244 (26h56m24.058905354s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019547526 (26h56m24.059120646s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019749592 (26h56m24.059322712s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019935007 (26h56m24.059508117s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:49.742054032 (26h56m25.781627152s) [upload=23710]: Next returned nil segment Sep 7 16:24:49.742070412 (26h56m25.781643522s) [upload=23710]: Waiting for error group managing segments... Sep 7 16:24:50.156729825 (26h56m26.196302945s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.179916844 (26h56m26.219489964s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.190874311 (26h56m26.230447431s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.206966775 (26h56m26.246539895s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.233097747 (26h56m26.272670867s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.259133567 (26h56m26.298706687s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.283376256 (26h56m26.322949376s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.294991472 (26h56m26.334564592s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.313999864 (26h56m26.353572985s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.327448961 (26h56m26.367022081s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.330061599 (26h56m26.369634719s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.335870615 (26h56m26.375443735s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350484780 (26h56m26.3900579s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350508866 (26h56m26.390081976s) [seg_pos={0 0}, upload=23710]: Segment reached optimal threshold of 13 pieces. Sep 7 16:24:50.350613436 (26h56m26.390186556s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350672970 (26h56m26.39024609s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350710802 (26h56m26.390283922s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:44398->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:24:50.350745478 (26h56m26.390318599s) [seg_pos={0 0}, upload=23710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:24:50.350811014 (26h56m26.390384135s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:24:50.350814852 (26h56m26.390387972s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:24:50.790876341 (26h56m26.830449461s) [upload=23710]: Done uploading segments... Sep 7 16:24:50.790895568 (26h56m26.830468688s) [upload=23710]: Upload finished. err: <nil> Duration: 7.270606625s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X......X..X.X......X.X.X..X...........X.....X.....X........] duration: 7.269561005s FAIL (2025-09-07 16:32:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:24:47.729359874 (26h56m23.768932994s) [upload=23710]: Starting upload Sep 7 16:24:47.729425129 (26h56m23.768998279s) [upload=23710]: Done starting upload Sep 7 16:24:47.729547583 (26h56m23.769120703s) [upload=23710]: Uploading segments... Sep 7 16:24:47.729893155 (26h56m23.769466275s) [upload=23710]: Got next segment. Inline: false Sep 7 16:24:47.729932419 (26h56m23.76950554s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:24:47.729936908 (26h56m23.769510028s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:24:48.017044215 (26h56m24.056617336s) [seg_pos={0 0}, upload=23710]: Begin upload segment... Sep 7 16:24:48.017099892 (26h56m24.056673012s) [seg_pos={0 0}, upload=23710]: Done begin upload segment. Sep 7 16:24:48.017123026 (26h56m24.056696146s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017169436 (26h56m24.056742556s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017486843 (26h56m24.057059963s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017740939 (26h56m24.057314059s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017838135 (26h56m24.057411255s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018088304 (26h56m24.057661414s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018122940 (26h56m24.05769606s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018402465 (26h56m24.057975585s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018482167 (26h56m24.058055287s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018681529 (26h56m24.058254649s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018912802 (26h56m24.058485912s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019116080 (26h56m24.0586892s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019332244 (26h56m24.058905354s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019547526 (26h56m24.059120646s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019749592 (26h56m24.059322712s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019935007 (26h56m24.059508117s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:49.742054032 (26h56m25.781627152s) [upload=23710]: Next returned nil segment Sep 7 16:24:49.742070412 (26h56m25.781643522s) [upload=23710]: Waiting for error group managing segments... Sep 7 16:24:50.156729825 (26h56m26.196302945s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.179916844 (26h56m26.219489964s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.190874311 (26h56m26.230447431s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.206966775 (26h56m26.246539895s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.233097747 (26h56m26.272670867s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.259133567 (26h56m26.298706687s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.283376256 (26h56m26.322949376s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.294991472 (26h56m26.334564592s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.313999864 (26h56m26.353572985s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.327448961 (26h56m26.367022081s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.330061599 (26h56m26.369634719s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.335870615 (26h56m26.375443735s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350484780 (26h56m26.3900579s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350508866 (26h56m26.390081976s) [seg_pos={0 0}, upload=23710]: Segment reached optimal threshold of 13 pieces. Sep 7 16:24:50.350613436 (26h56m26.390186556s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350672970 (26h56m26.39024609s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350710802 (26h56m26.390283922s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:44398->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:24:50.350745478 (26h56m26.390318599s) [seg_pos={0 0}, upload=23710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:24:50.350811014 (26h56m26.390384135s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:24:50.350814852 (26h56m26.390387972s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:24:50.790876341 (26h56m26.830449461s) [upload=23710]: Done uploading segments... Sep 7 16:24:50.790895568 (26h56m26.830468688s) [upload=23710]: Upload finished. err: <nil> Duration: 7.269587114s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X......X..X.X......X.X.X..X...........X.....X.....X.......] duration: 11.545038204s FAIL (2025-09-07 16:30:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:24:47.729359874 (26h56m23.768932994s) [upload=23710]: Starting upload Sep 7 16:24:47.729425129 (26h56m23.768998279s) [upload=23710]: Done starting upload Sep 7 16:24:47.729547583 (26h56m23.769120703s) [upload=23710]: Uploading segments... Sep 7 16:24:47.729893155 (26h56m23.769466275s) [upload=23710]: Got next segment. Inline: false Sep 7 16:24:47.729932419 (26h56m23.76950554s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:24:47.729936908 (26h56m23.769510028s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:24:48.017044215 (26h56m24.056617336s) [seg_pos={0 0}, upload=23710]: Begin upload segment... Sep 7 16:24:48.017099892 (26h56m24.056673012s) [seg_pos={0 0}, upload=23710]: Done begin upload segment. Sep 7 16:24:48.017123026 (26h56m24.056696146s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017169436 (26h56m24.056742556s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017486843 (26h56m24.057059963s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017740939 (26h56m24.057314059s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017838135 (26h56m24.057411255s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018088304 (26h56m24.057661414s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018122940 (26h56m24.05769606s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018402465 (26h56m24.057975585s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018482167 (26h56m24.058055287s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018681529 (26h56m24.058254649s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018912802 (26h56m24.058485912s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019116080 (26h56m24.0586892s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019332244 (26h56m24.058905354s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019547526 (26h56m24.059120646s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019749592 (26h56m24.059322712s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019935007 (26h56m24.059508117s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:49.742054032 (26h56m25.781627152s) [upload=23710]: Next returned nil segment Sep 7 16:24:49.742070412 (26h56m25.781643522s) [upload=23710]: Waiting for error group managing segments... Sep 7 16:24:50.156729825 (26h56m26.196302945s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.179916844 (26h56m26.219489964s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.190874311 (26h56m26.230447431s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.206966775 (26h56m26.246539895s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.233097747 (26h56m26.272670867s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.259133567 (26h56m26.298706687s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.283376256 (26h56m26.322949376s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.294991472 (26h56m26.334564592s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.313999864 (26h56m26.353572985s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.327448961 (26h56m26.367022081s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.330061599 (26h56m26.369634719s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.335870615 (26h56m26.375443735s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350484780 (26h56m26.3900579s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350508866 (26h56m26.390081976s) [seg_pos={0 0}, upload=23710]: Segment reached optimal threshold of 13 pieces. Sep 7 16:24:50.350613436 (26h56m26.390186556s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350672970 (26h56m26.39024609s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350710802 (26h56m26.390283922s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:44398->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:24:50.350745478 (26h56m26.390318599s) [seg_pos={0 0}, upload=23710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:24:50.350811014 (26h56m26.390384135s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:24:50.350814852 (26h56m26.390387972s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:24:50.790876341 (26h56m26.830449461s) [upload=23710]: Done uploading segments... Sep 7 16:24:50.790895568 (26h56m26.830468688s) [upload=23710]: Upload finished. err: <nil> Duration: 11.545066388s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X......X..X.X......X.X.X..X...........X.....X.....X......] duration: 6.376968363s FAIL (2025-09-07 16:29:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 16:24:47.729359874 (26h56m23.768932994s) [upload=23710]: Starting upload Sep 7 16:24:47.729425129 (26h56m23.768998279s) [upload=23710]: Done starting upload Sep 7 16:24:47.729547583 (26h56m23.769120703s) [upload=23710]: Uploading segments... Sep 7 16:24:47.729893155 (26h56m23.769466275s) [upload=23710]: Got next segment. Inline: false Sep 7 16:24:47.729932419 (26h56m23.76950554s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 16:24:47.729936908 (26h56m23.769510028s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 16:24:48.017044215 (26h56m24.056617336s) [seg_pos={0 0}, upload=23710]: Begin upload segment... Sep 7 16:24:48.017099892 (26h56m24.056673012s) [seg_pos={0 0}, upload=23710]: Done begin upload segment. Sep 7 16:24:48.017123026 (26h56m24.056696146s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017169436 (26h56m24.056742556s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017486843 (26h56m24.057059963s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017740939 (26h56m24.057314059s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.017838135 (26h56m24.057411255s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018088304 (26h56m24.057661414s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018122940 (26h56m24.05769606s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018402465 (26h56m24.057975585s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018482167 (26h56m24.058055287s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018681529 (26h56m24.058254649s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.018912802 (26h56m24.058485912s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019116080 (26h56m24.0586892s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019332244 (26h56m24.058905354s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019547526 (26h56m24.059120646s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019749592 (26h56m24.059322712s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:48.019935007 (26h56m24.059508117s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Uploading piece... Sep 7 16:24:49.742054032 (26h56m25.781627152s) [upload=23710]: Next returned nil segment Sep 7 16:24:49.742070412 (26h56m25.781643522s) [upload=23710]: Waiting for error group managing segments... Sep 7 16:24:50.156729825 (26h56m26.196302945s) [noise=true , address=45.91.116.12:4227 , piece_id=R3C7FAHCLYH76YLP7G4TIEE7MIIMSEWBPB5W4WR5DZN3B4WXCKWQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.179916844 (26h56m26.219489964s) [noise=true , address=45.91.116.10:4082 , piece_id=POCW5BYB4MKDK55CCVTG7C2N6KUMKRBGNAXAUPUAQO44PXQWZIBQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.190874311 (26h56m26.230447431s) [noise=true , address=45.91.116.4:4937 , piece_id=U7SX4BDDMP5T45HGZSGSXHQFPNQT5ERTJUR2SMTPQMSZT5KF6JIA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.206966775 (26h56m26.246539895s) [noise=true , address=45.91.116.6:4975 , piece_id=SKFSA3L3RMKBJQZYHH5MBEIXNG4F2RYHF4PIQMMUE34462XPGJYQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.233097747 (26h56m26.272670867s) [noise=true , address=45.91.116.14:4191 , piece_id=GFWAYYPL6B4WDVSO3O57MQ2MVF44WUALLAUAKWT56GTCCJ6DWH3Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.259133567 (26h56m26.298706687s) [noise=true , address=77.90.31.12:4589 , piece_id=WJ4J3OTLBINO5ZN37ZEV36JOIPACCMYAJQ6V4IGMXRBTWGKPF47A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.283376256 (26h56m26.322949376s) [noise=true , address=77.90.31.17:4771 , piece_id=FG6662I2VMJF4WPOLPW3VFMQYSPK4X3KGYILSFNDIM3SBISA3C2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.294991472 (26h56m26.334564592s) [noise=true , address=45.91.116.6:4974 , piece_id=2HXM7MFFHHCVKYWGHOZQABHJMOVCXQMWQMG6D2T2XMGG6DFVOILQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.313999864 (26h56m26.353572985s) [noise=true , address=45.91.116.10:4081 , piece_id=I34RGSBWTCMPH2SJPQNA2ISGIZZH62VLNL672A5VV5P4SYNPHUDA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.327448961 (26h56m26.367022081s) [noise=true , address=45.91.116.8:4047 , piece_id=7GYMHPYNRTZWSAW3PBMJ5JFBGOBTYOSFOF3NDCQCORJRPYJKDR2Q, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.330061599 (26h56m26.369634719s) [noise=true , address=45.91.116.11:4117 , piece_id=C6YFO2UJNRIYBLW75VLUXDV3TELDHG27ZYKYKJ22DEGNHVE7BTPQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.335870615 (26h56m26.375443735s) [noise=true , address=45.91.116.11:4119 , piece_id=FDZCKZ3EUANYTAJC474LNSSLJ55AVKXV4HLQSJ5V4UGPWK7C6AFA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350484780 (26h56m26.3900579s) [noise=true , address=45.91.116.9:4262 , piece_id=7VX5VLEBUNH7IGYBRHVP2C343LMBTCCZ6PX6FO5RQK5UGWVJRWSA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: <nil> Sep 7 16:24:50.350508866 (26h56m26.390081976s) [seg_pos={0 0}, upload=23710]: Segment reached optimal threshold of 13 pieces. Sep 7 16:24:50.350613436 (26h56m26.390186556s) [noise=true , address=77.90.31.15:4697 , piece_id=27BJYZC3NI3ICVVB33SOHGJJ662AC3PK2E63AWEV42BCUCTY4JOQ, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350672970 (26h56m26.39024609s) [noise=true , address=77.90.31.15:4699 , piece_id=4LX72UB6I6JPGT34E4I7ESKLEA3YRFIP224OICARZJ2HFK43JYUA, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF Sep 7 16:24:50.350710802 (26h56m26.390283922s) [noise=true , address=77.90.31.21:4877 , piece_id=CLVCZR2C4HIOUIXPBV7FGIQIQ5P2SL6MKCAWP2FSXY7EJ56L7R7A, seg_pos={0 0}, upload=23710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:44398->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Sep 7 16:24:50.350745478 (26h56m26.390318599s) [seg_pos={0 0}, upload=23710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 16:24:50.350811014 (26h56m26.390384135s) [upload=23710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 16:24:50.350814852 (26h56m26.390387972s) [upload=23710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 16:24:50.790876341 (26h56m26.830449461s) [upload=23710]: Done uploading segments... Sep 7 16:24:50.790895568 (26h56m26.830468688s) [upload=23710]: Upload finished. err: <nil> Duration: 6.376990175s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)