Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.662938736s FAIL (2025-07-18 01:54:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.662969304s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 5.679550597s FAIL (2025-07-18 01:54:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.679581237s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 5.769028244s FAIL (2025-07-18 01:53:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.769112645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.860725579s FAIL (2025-07-18 01:52:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.860749325s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 5.976963705s FAIL (2025-07-18 01:52:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.976992039s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 6.182155096s FAIL (2025-07-18 01:52:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 6.182179964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.941480264s FAIL (2025-07-18 01:52:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.941502317s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 5.785084675s FAIL (2025-07-18 01:51:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.78511298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 5.755290175s FAIL (2025-07-18 01:51:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.755317188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.X.........................................................] duration: 5.956395671s FAIL (2025-07-18 01:50:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.956439585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 5.760612518s FAIL (2025-07-18 01:50:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.760647974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.X........................................................] duration: 6.008437755s FAIL (2025-07-18 01:50:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.008460498s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 5.717036077s FAIL (2025-07-18 01:50:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.717075242s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.X.......................................................] duration: 5.948970514s FAIL (2025-07-18 01:49:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.948996102s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 5.848255871s FAIL (2025-07-18 01:49:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.848278334s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.X......................................................] duration: 5.884545293s FAIL (2025-07-18 01:49:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.884618913s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.X.....................................................] duration: 5.938065552s FAIL (2025-07-18 01:48:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.938083616s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 5.921044337s FAIL (2025-07-18 01:48:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.92106741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.X....................................................] duration: 5.839598215s FAIL (2025-07-18 01:48:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.839626969s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 5.710982564s FAIL (2025-07-18 01:47:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.711011059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.X...................................................] duration: 5.942127314s FAIL (2025-07-18 01:47:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.942151309s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 5.846267356s FAIL (2025-07-18 01:46:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.846296461s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.X..................................................] duration: 5.858225415s FAIL (2025-07-18 01:46:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.858258038s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 5.893647497s FAIL (2025-07-18 01:46:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.893681392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.X.................................................] duration: 5.888080629s FAIL (2025-07-18 01:45:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.888108293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 5.962277461s FAIL (2025-07-18 01:45:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.962307789s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.X................................................] duration: 6.124057764s FAIL (2025-07-18 01:45:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.124091106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 5.643067297s FAIL (2025-07-18 01:44:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.643102043s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 5.838806124s FAIL (2025-07-18 01:44:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.838834957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.X...............................................] duration: 5.925743289s FAIL (2025-07-18 01:43:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.92577s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 5.737598605s FAIL (2025-07-18 01:43:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.737626579s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.X..............................................] duration: 5.883666956s FAIL (2025-07-18 01:43:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.883698947s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.X.............................................] duration: 5.973603502s FAIL (2025-07-18 01:42:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.973633968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 5.88709893s FAIL (2025-07-18 01:42:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.887130079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.X............................................] duration: 5.994525501s FAIL (2025-07-18 01:41:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.994552441s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.X...........................................] duration: 5.865819891s FAIL (2025-07-18 01:41:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.865848596s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 5.756343828s FAIL (2025-07-18 01:41:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.756390396s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 5.865932674s FAIL (2025-07-18 01:40:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.865957982s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.X..........................................] duration: 5.888048809s FAIL (2025-07-18 01:40:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.888076272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.X.........................................] duration: 6.031713183s FAIL (2025-07-18 01:40:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.031740425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 5.988104074s FAIL (2025-07-18 01:40:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.988134162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.X........................................] duration: 5.917758013s FAIL (2025-07-18 01:39:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.917781277s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 5.823318361s FAIL (2025-07-18 01:39:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.823349781s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.X.......................................] duration: 5.964458033s FAIL (2025-07-18 01:38:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.964482359s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 5.649357994s FAIL (2025-07-18 01:38:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.649386699s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.X......................................] duration: 5.940519854s FAIL (2025-07-18 01:38:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.940547786s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.X.....................................] duration: 5.947845949s FAIL (2025-07-18 01:37:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.947876338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 6.148815841s FAIL (2025-07-18 01:37:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 6.148843535s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.X....................................] duration: 5.925431332s FAIL (2025-07-18 01:37:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.925469164s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 5.770078496s FAIL (2025-07-18 01:36:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.77010636s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)