Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.828104876s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:37:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.828132719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.824028489s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:36:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.824043376s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.854226731s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:35:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.854254905s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.942674485s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:35:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.942704682s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.826388792s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:34:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.826477111s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.751165404s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:33:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.751196622s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.778435266s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:33:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.77848531s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.76537024s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:32:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.765410134s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.833176599s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:31:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.833206215s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.794266076s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:30:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.79429462s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.981548247s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:30:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.981655562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.73015245s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:29:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.730171776s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.924016446s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:28:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.924044859s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.988245512s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:28:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.988274577s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.014712878s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:27:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.014731884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.241647738s trace: https://jaeger.dev.storj.io/trace/1587802e61ecd031 FAIL (2025-11-27 17:27:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 17:27:10.072664359 (441h23m8.62413481s) [upload=388675]: Starting upload Nov 27 17:27:10.072747678 (441h23m8.624218129s) [upload=388675]: Done starting upload Nov 27 17:27:10.072818543 (441h23m8.624289004s) [upload=388675]: Uploading segments... Nov 27 17:27:10.072984900 (441h23m8.624455361s) [upload=388675]: Got next segment. Inline: false Nov 27 17:27:10.073021660 (441h23m8.624492111s) [upload=388675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 17:27:10.073026449 (441h23m8.62449691s) [upload=388675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 17:27:10.318208353 (441h23m8.869678814s) [seg_pos={0 0}, upload=388675]: Begin upload segment... Nov 27 17:27:10.318258930 (441h23m8.869729391s) [seg_pos={0 0}, upload=388675]: Done begin upload segment. Nov 27 17:27:10.318304116 (441h23m8.869774567s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318357076 (441h23m8.869827527s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.318992427 (441h23m8.870462878s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319224368 (441h23m8.870694829s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319302768 (441h23m8.870773229s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319596166 (441h23m8.871066617s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319624160 (441h23m8.871094621s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.319994385 (441h23m8.871464846s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320080448 (441h23m8.871550909s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320351765 (441h23m8.871822226s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.320734384 (441h23m8.872204845s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321107855 (441h23m8.872578306s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321455327 (441h23m8.872925788s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.321913970 (441h23m8.873384431s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322541485 (441h23m8.874011946s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:10.322819584 (441h23m8.874290045s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Uploading piece... Nov 27 17:27:12.007117702 (441h23m10.558588163s) [upload=388675]: Next returned nil segment Nov 27 17:27:12.007140305 (441h23m10.558610766s) [upload=388675]: Waiting for error group managing segments... Nov 27 17:27:12.335127408 (441h23m10.886597869s) [noise=true , address=77.90.31.11:4554 , piece_id=PB6QYCJPYZ3UTZRJQ5PN2XB6M22DMPSPVPXTEULM5N7T6EVAEQWQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.378742219 (441h23m10.93021268s) [noise=true , address=77.90.31.10:4518 , piece_id=BGSYCEXEMAJSO274QCV6F2FYSFUEPKD75MD6QXXZVY3SKCBYO6AA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:12.394603502 (441h23m10.946073963s) [noise=true , address=77.90.31.17:4770 , piece_id=NC335VXVNPCM26E7RLA6AT2DE3LCWVEGYX5QOFIX7UTMM6JVCVKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.255496852 (441h23m13.806967313s) [noise=true , address=145.79.227.9:4367 , piece_id=KMO3EUHAG3U4ULFO3JSX2S6OOLN4N3YPM7ZNXHGZ72WQZOGBXIZA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.338243418 (441h23m13.889713879s) [noise=true , address=145.79.227.26:4654 , piece_id=A6H7IQZJASQNP5L6TKNA2OL4BBZX5IAU2FKDLDHTIGTRTFONAE3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:15.998331817 (441h23m14.549802278s) [noise=true , address=145.79.227.17:4971 , piece_id=42DX5OEU33PDTM24U6WLZX2ID6N2JKS4JJIQQOJORMNTMMYHZGLA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.060334221 (441h23m15.611804682s) [noise=true , address=145.79.227.32:4836 , piece_id=HVRXUNMWKYUY5ZGWN6PK6NU76WNX3MOSNE2SRPXOKAEHYU4LQSVA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:17.416883772 (441h23m15.968354233s) [noise=true , address=145.79.227.41:4086 , piece_id=VTALII7R5CDI5ET2H5Y5TYFBHYLAEZNL4PUHGBZPD5E427EPPY3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.279896276 (441h23m20.831366737s) [noise=true , address=145.79.227.22:4511 , piece_id=IPXE4QHJ3HVWYZKBT4E4Y76LU6YSEHFDUM5WTFASX4TMQF54YIQQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:22.938703255 (441h23m21.490173716s) [noise=true , address=145.79.227.30:4762 , piece_id=6TBZ7R2PRIMV5ZEUV7UK6WTO5CVBZUJESP3ILEXQTMZQAJQR22KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:27.153137244 (441h23m25.704607705s) [noise=true , address=145.79.227.23:4547 , piece_id=EX374TABIYYQLHJFB6GQMAGBUZLFRSX4UVMOSDEIKBMHYWLM5G4A, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:29.168972153 (441h23m27.720442614s) [noise=true , address=145.79.227.27:4690 , piece_id=Q4GKKPEETACUYA55CIQIF2SQYLUM63L5FAWCBVP5CJ2ZEUJLIG3Q, seg_pos={0 0}, upload=388675]: Done uploading piece. err: <nil> Nov 27 17:27:39.372250644 (441h23m37.923721115s) [noise=true , address=145.79.227.15:4875 , piece_id=VFL3YNDG5IJOUOMS6PUL5DJNMTNK4CKFGGT5YX3G32DDKQILIGSQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38984->145.79.227.15:4875: use of closed network connection Nov 27 17:27:39.372362637 (441h23m37.923833088s) [noise=true , address=145.79.227.42:4121 , piece_id=HATYBV7PV7RXHD56OTRMOLYDHOWC5RJWXOEQ3NMUVEQVK7JH2QKQ, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37042->145.79.227.42:4121: use of closed network connection Nov 27 17:27:39.372395189 (441h23m37.923865641s) [noise=true , address=145.79.227.24:4583 , piece_id=YZSDJ2U5CZNIGFMFN6UDBAR6P66NHZ5536HKX4FZOQJKUPBOQ2KA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55742->145.79.227.24:4583: use of closed network connection Nov 27 17:27:39.372514898 (441h23m37.923985359s) [noise=true , address=145.79.227.11:4730 , piece_id=FJBPEH7XVZJLHOMNTIQIFH7N6ZVHNVKLXHMTLEVBP2PFAYNYYPMA, seg_pos={0 0}, upload=388675]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47342->145.79.227.11:4730: use of closed network connection Nov 27 17:27:39.372552219 (441h23m37.92402268s) [seg_pos={0 0}, upload=388675]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 17:27:39.612397282 (441h23m38.163867743s) [upload=388675]: Done uploading segments... Nov 27 17:27:39.612443080 (441h23m38.163913551s) [upload=388675]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.241682174s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 76.48333ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1118720b4e827676 FAIL (2025-11-27 16:13:47 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:53938->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 76.587147ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 79.355449ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1118720b4e827676 FAIL (2025-11-27 16:13:07 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:53938->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 79.391899ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 71.861299ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1118720b4e827676 FAIL (2025-11-27 16:11:57 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:53938->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 71.940289ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 84.331235ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1118720b4e827676 FAIL (2025-11-27 16:11:33 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:53938->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 84.356252ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 87.770163ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1118720b4e827676 FAIL (2025-11-27 16:10:38 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:53938->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 87.803868ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 76.683422ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1118720b4e827676 FAIL (2025-11-27 16:09:57 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:53938->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 76.715242ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 55.996229ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1118720b4e827676 FAIL (2025-11-27 16:09:12 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:53938->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 56.0236ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 79.225442ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1118720b4e827676 FAIL (2025-11-27 16:08:15 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:53938->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 79.258153ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 75.120164ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1118720b4e827676 FAIL (2025-11-27 16:07:05 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:53938->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 75.153026ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 92.029162ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1118720b4e827676 FAIL (2025-11-27 16:06:25 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:53938->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 92.066493ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 68.829826ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1118720b4e827676 FAIL (2025-11-27 16:05:56 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:53938->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 68.866515ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 234.695789ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1118720b4e827676 FAIL (2025-11-27 16:04:49 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:53938->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 234.721459ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 68.938483ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1118720b4e827676 FAIL (2025-11-27 16:04:16 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:53938->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 68.965956ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 81.572088ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1118720b4e827676 FAIL (2025-11-27 16:03:29 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:53938->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 81.596925ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 242.633073ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/1118720b4e827676 FAIL (2025-11-27 16:02:19 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:53938->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 242.680172ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.497267769s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:57:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.497299801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 4.351290267s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:56:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.351320634s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.417583204s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:55:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.417620163s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.435066396s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:54:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.435102855s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.454720912s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:52:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454750489s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.702955382s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:52:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.703059329s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.485016812s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:51:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.485051589s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.478448369s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:51:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.478478195s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.439252827s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:50:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.439289978s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.472582286s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:49:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.472613344s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.373856603s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:48:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.373884605s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.815841002s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:48:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.815957604s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.545651405s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:47:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.545677505s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.566366086s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:46:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.566392707s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.461546235s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:45:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.461583185s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.447917169s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:44:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.447945625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.52239714s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:44:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.522425804s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 4.415679024s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:43:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.415712278s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.534247269s trace: https://jaeger.dev.storj.io/trace/352f8a9c51a1e1c1 FAIL (2025-11-27 15:42:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.534268962s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)